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ABSTRACT 

project Information Packages (PIPs) , kits containing 
directions for the replication of exemplary remedial programs, were 
field tested in a nuEfber of school districts. Based on curriculum 
analysis and site visits PIPs were found to induce projects which 
were adequate copies of what was packaged; however, what was packaged 
was not sufficient to implement the same curriculum across sites. 
Particular emphasis was given to the statistical procedures used in 
comparing the extent to which the PIPs were implemented in the test 
sites and in determining gains on the Metropolitan Achievement Tests 
(MAT) . The MAT and the associated norm-referenced a *alysis were 
closely examined. The assumption that PIPs, if fully implemented in 
appropriate school settings, would cause an increase in MAT scores 
was found to be false. This relationship was also not detectable at 
all grade levels. Analysis revealed that teacher responsiveness, 
rather than PIP implementation, was more often associated with gains 
in test scores. Suggestions for future evaluation methods and the 
manual of procedures for PIP testing are included. (GDC) 
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PREFACE 



In my opinion, this technical report is somewhat different from 
the usual Office of Education evaluation product. 

The differences are not so much due to differences in the questions 
we asked and the difficulties we faced. The differences are due to our 
idea of what constitutes a relevant evaluation of an educational product 
that is supposed to raise norm-referenced, standardized achievement 
test scores. 

Enough evidence had accumulated to make us doubt the reasonableness 
of the assumption that available standardized achievement tests are 
sensitive to what most compensatory reading teachers are doing in class. 
Thus, we sought to document facts that would make it plausible that the 
treatment received by s.idents would teach them what to answer on the 
test selected as the outcome mersure. First, this entailed finding out 
what was actually done (not merely what was supposed to be done) ; and 
second, it entailed an analysis of the connection between what was done 
and what would teach the items. 

Most evaluation products address these two points in an abstract 
way referring to iter?* domains and teaching objectives, instead of 
specific items and activities. We bave attempted to do a more detailed 
analysis. The reader will judge how successful we were. 

This report also differs somewhat from many Office of Education 
evaluation products in its technical approach. Where possible, we have 
not calculated statistical hypothesis tests which are meaningless in 
this type of evaluation. VJe tried to create successful inductive 
arguments based on reported facts, not based on a nonexistent sampling 
scheme. We also attempted to directly measure our variables for formal 
analysis. This avoided the necessity of making post-hoc scales for some 
important variables. For example, we had our site visit staff assess 
implementation based on their experience; no further scaling was done. 
Our regression equations were aimed at modeling the bivariate distribution 
of pretest and posttest scores, as opposed to the distribution of posttest 
scores given pretest scores. The reader will judge how successful these 
techinques were. 

I attempted to include in this report enough information about our 
procedures and analyses so that the reader could understand the bases 
for our conclusions, draw his own conclusions, or carry out the activities 
himself in another study. The majority of readers interested in the PIP 
field test and its outcomes are not concerned with the details of the 
steps we took in our evaluation. These readers are referred to Volume I, 
the Summary Report. 
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The original six PIPs, ^^eveloped by RMC and tested in the field 
during the school years 1974-1976 are the subject of the evaluation 
described in this report. These original PIPs are no longer in use; 
they were thoroughly revised by our subcontractor, RMC, on the basis 
of first-year evaluation findings. 

This report owes its quality to the efforts of the project staff. 
Our evidence on whether the PIP projects should influence outcome scores 
was analyzed by Arlene B. Tennenbaum md Christine Miller, assisted by 
Christine Padilla* Many technical problems associated with our analysis 
were investigated by David Kaskowitz. 

The high validity of our field work is due to the conscientious 
work of our field visit staff: Dorothy Booth, Jay Cross, Cassandra Duarte, 
Phillip Giesen, Georgia Gillis, and Margaret Needels. They actively 
p -xticipaited in all phases of the project. 

Encoding of dat£ and preparation of raw data files was supervised 
by Bert Laurence and Bill Lambert. The creation of analysis files and 
execution of analyses were supervised by George Black. George Byrd, 
Jerry Kauffman, Pat McCall, John Roilin, and Roy Sutton all helped 
program analyses. Quality control procedures for machine -re ad able data 
were supervised by Elizabeth Milrod. 

The labor of report writing was shared by Trudy Nio (Chapter 1), 
David Kaskowitz (Chapter 3), Georgia Gillis (Chapter 4), Marian Stearns 
anc^ Christine Miller (Chapter 5), and mysBlf. Elizabeth Milrod, assisted 
by Nancy Craig, who was project secretary, and Christine Padilla 
coordinated our efforts. 

Marian Scearns vas project director and author of Volume I of this 
report. Ativ Bezc'ek vas our project officer. 



Charles R. Norwood 
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SUM>U\RY 



EVALUATION OF THE FIELD TEST OF PROJECT INFORJ^IATION PACKAGES (PIPs) 



Purpose of the Study 

In its continuing search for successful means by which to disseminate 
exemplary education projects, the U.S. Office of Education supported the 
development of six Project Informatioa Packages (PIPs) in 1973, The PIPs 
were designed to provide "how to" information and instructions to facili- 
tate the installation and implementation of exemplary compensatory 
reading and mathematics projects in new school districts with a minimum 
of technical assistance. The projects selected for packaging passed 
criteria of effectiveness with respect to reading and mathematics skills^ 
as well as meeting other criteria. They were reviewed and passed as 
exemplary projects by the Education Division Joint Dissemination Review 
Panel. Four of the projects were for elementary school students and two 
were for middle or secondary school students. Five of the six projects, 
originally developed by local education agencies (LEAs) with ESEA Title I 
funds, were "pull-out" projects. These projects supplemented the regular 
school reading or mathematics programs rather than replacing them; they 
served a special target group of students, and required additional space 
and teaching staff. The sixth project, for junior high school students, 
originally developed by a local education agency wi'^U other federal funds, 
was not a pull-out project but, rather, served^ all students in a specified 
grade and required regular classroom teachers to make changes in their 
instructional methods. 

The central principle assumed in developing the PIPs (the "replication" 
principle) was that, if the antecedent conditions of the effective instruc- 
tional project could be established in a new site, the project would be 
reproduced in the new site, and would again prove effective in terms of 
student achievement gains. In addition, two other assumptions were made. 
First, it was assumed necessary to match the setting of the replicating 
site with the setting of the original, successful site. This was to be 
accomplished by providing information to potential project adopter sites 
about the original projects and their settings in an Analysis and Selection 
Kit (ASK). Districts interested in replicating a packaged project would 
use the ASK in selecting an appropriate project that matched local condi- 
tions. Second, it was assumed that project management was the key to 
replicating the original conditionr^. Given this assumption, the PIPs 
highlighted the importance of a dynamic, experienced project director who 
had responsibility for orienting others in the school district, hiring 
staff and providing for inservice training and mobilizing the resources 
necessary to establish the antecedent conditions for effective instruction. 
Information provided in the PIPs was management-oriented to help project 
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directors and teachers set up conditions for the original instructional 
program to be recreated; the packages specified requirements for space, 
qualified staff, materials and equipment, student selection, scheduling, 
record keeping, and the like. The PIPs differed in the' amount of infor- 
mation they provided about the actual instructional program. All but 
one did not describe in detail the teaching/ learning episode, classroom 
interaction, or sequences of events within the instructional program. 
Nor did the PIPs describe the uses of each of the recommemded curriculum 
materials, since it was assumed that the appropriate events would follow, 
given the appropriate mix of resources and the specified teaching staff. 

In fiscal year 1975, PIPs were tried out in a number of sites across 
the country. Under Section 306 of ESCA Title III, grants were awarded 
to 19 sites for the purpose of implementing one of ths exemplary projects 
via a PIP at each site. To assess the feasibility of replicating 
successful projects via packages as a way of improving reading and 
mathematics skills of disadvantaged children, a contract was awarded to 
Stanford Research Institute in June 1974. The two-year study was to 
examine the implementation of the packaged projects in the tryout sites 
and to focus on the following questions: 

• Are local education agencies motivated to adopt a packaged 
project? 

• Can exemplary projects be implemented in new sites via the 
PIPs? Where implementation problems are due to faults in 
the packages, can reasonable modifications be recommended? 

• What functions and in what amount is technical assistance 
required? If considerable technical assistance is required, 
can the packages b'2 made more autonomous? 

• Are the projects, implemented via the PIPs, effective in 
improving student achievement? 

• What are the effects of the projects on student attitudes? 
Are the projects acceptable to the local education agency, 
to teachers, parents, and the community? 

• What is the cost of implementing the projects? 

The first-year study objec»-ives were to examine the project adoption 
and installation processes by documenting the discrepancies in each of 
the 19 sites from PvMC expectations and PIP specifications, determining 
che usefulness of the PIPs for guiding project implementation, and 
determining the soundness of the principles and assumptions upon which 
the PIPs were developed. Also of concerri during the initial year were 
the identification of implementation problems encountered by the tryout 
sites and recommendacions about how thi- pack;-ges might be revised i:. 
light of the problems identified. RMC va£ Cne subcontractor responsible 
for the latter formative evaluation tasks. 
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The second-year study objectives were to investigate the impact of 
the projects on student achievement, to explain differences from expected 
outcomes, and to explore the participating school districts* intentions 
for continuing the projects in school year 1976-77 when Title III, 
Section 306 money was no longer available. RMC created a new ASK .-.nd 
revised the PIPs based on results of the first year study. 

The Study Approach 

The central concept of the ASK/PIP dissemination strategy that 
of replication: In the right community, the PIP with little or no 
technical assistance, plus a good faith effort on the pait of qualified 
staff, will replicate the successes of the original project. 

To test this concept the first year was devoted to assessing the 
degree of project instal lation, and the effectiveness of the pr incip les 
used to select and package the exemplary projects to be reproduced. 
Therefore, the first year's major evaluation strategy was to compare 
projects with PIP specifications. 

During the first year, five visits were made to each of the 19 field- 
test sites to observe the project, to interview project and nonproject 
personnel involved in installation, and to conduct pre- and post-student 
testing. Although the first year of the evaluation focused on prjject 
installation rather than impact on participating students, both standard- 
ized achievement tests and attitude surveys were acministered to a minimal 
sample of participating students to get a sense of likely effects and to 
make sure that the implementation year was not disuptive in terms of 
student achievement and attitudes. Observations of the projects during 
the site visits ^ere used to determine the degree of implementation of 
the specified elements in the instructional environment. Interviews 
were conducted with administrative and instructional staff members to 
learn about the installation process, to determine the causes of imple- 
mentation problems, project modifications, and deviations from PIP 
specifications to determine acceptability of the packaged projects and 
the like. Informal contacts were also made with parents of children who 
were participants to determine their reaction to the project. 

In addition to the site visits, contact report forms were used 
throughout the year by project staff at the original c.'nd tryout project 
sites and by government and evaluation staff to report telephone conver- 
sations, visits, and other contacts in which assistance or clarification 
was requested, offered, or obtained. Finally, instructional staff 
questionnaires and administrative staff questionnaires were administered 
to assess staff attitudes toward the PIP and the local project, and 
resource/cost questionnaires were administered to determine the resources 
and associated costs of project installation. 

The focus of the second year study was on project effectiveness as 
measured by achievement test gains. 
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To ensure that the instructional programs described in the PIPs were 
well understood, a conference was held in Washington, D.C. At this 
conference PIP project directors and the project directors at the original 
sites were brought together. RMC staff members who developed the PIPs 
were also present. This meeting provided the PIP project staff with 
direct confirmation, or disconf irmation, of the practices which were 
in.*:^aded to be communicated by the PIPs. 

After this conference, our evaluation activities in the field were 
essentially similar to tho'-;e of the first year, except that in the second 
year we tested all children, not just a sample as in the first year, and 
we collected lesson plans and information on project curricula from 
teachers in the site visit sample. 

In the second year we also found it necessary to add some special 
studies to our evaluation activities. 

Results of the First Year Study 

The results of our first year study, reported in Stearns (1976), 
were generally favorable. Projects were implemented to the degree of 
specificity required by the PIPs, and this was done with relatively 
minor technical assistance. Student attitudes were not adversely 
impacted by project participation, and most project staff were 
enthusiastic . 

However, there was little evidence that the PIP projects raised 
reading scores on the Metropolitan Achievement Test (1970) to the extent 
it was expected the original projects would have. We could see no reason 
why this result would not reoccur in the second year. Consequently we* 
conducted some special studies of the analytic foundations of the PIP 
criterion of success. 



Results of the Second Year Study 
Special Studies 

The special studies examined several assumptions implicit in our 
concept of what would be evidence for PIP success 4 Our work may be 
convei>iently divided into two areas. The first is an examination of the 
properties of the MAT as applied to the PIP evaluation (a test of the 
replication principle). The second was our examination of the properties 
of our statistical procedures as applied to the MAT. 

The PIP replication principle is rooted in widely held beliefs about 
the worth and properties of reading comprehension tests of the type 
represented by the MAT. One of these beliefs is that, except for statis- 
tical features, most nationally normed tests are essentially the same. 
For example, OE bas funded the Anchor Test Study in which the MAT and 
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seven other tests were treated as equivalent tests of reading comprehension. 
Therefore, the replication principle does not assert that th*? FTi* projects 
will raise test scores on only the tests which the origin3tii.v> projects 
used. It is usually suggested chat such projects would not be worth 
disseminating, if their effects were so specific. 

These beliefs have not been unchallenged., [See and Green 

(in press) for papers on both sides of this conurovcrs/. ] We examined 
the issue of how specific the effects of the Plr projects would be, 
based on the premise that in projects wht^.ro the correct PIP instructional 
style is used, learning [,ains would .follow fron* i reasonable length of 
exposure to the PIP curriculum. We also assun-i-id that if a reading 
comprehension test is relevant to the Plr^ curriculum these gains would 
be detectable . 

We therefore examined the PIPs to identify the instructional style 
that was specified. We fouad that very little information was provided 
on the projects* philosophy of learning and instructional practices for 
teaching children. It is fair to say that, overall, the packages described 
project installation primarily from a Project Director's point of view; 
they did not describe instruction from the point of view of a teacher 
who had never implemented the project before. 

That the PI? did not presume to teach how to teach was no oversight 
on the part of the package designers. The PIPs attempted to establish 
the preconditions for obtaining achievement gains by focusing on placement 
of projects in coc.nunities with appropriate resources (through the ASK), 
by focusing on project and classroom management, and by hiring staff who 
had qualifications similar to those at the original site. 

Therefore we determined to analyze the curriculum materials which 
were used in tne projects for their relevance to MAT Reading. We found 
th<it projects using a given PIP did not use the same teaching materials, 
that the materials used were not very different from the regular class- 
room materials, and that project niaterials generally were not especially 
relevant to reading comprehension tested by the MAT. (At the first and 
second, grades, content of the curriculum materials and the MAT Reading 
subtest corresponded somewhat more closely.) In summary, it would be 
surprising if the projects, however well implemented, showed remarkable 
gains on MAT Reading. And since the PIP curriculum was not very different 
from the regular curriculum, what gains were observed might not be due 
to the PIP project. A brief examination of the tests used to validate 
the originating projects showed that the original tests were generally 
more suited to the curricula. 

The investigation' of the properties of our statistical procedure as 
applied to the MAT yielded sonle interesting formal results, as well as a 
better understanding of the implications of accepting the MAT norms as 
longitudinally valid. We concluded the assumption that the MAT norms 
are longitudinally valid should not be uncritically accepted. Assuming 
that the MAT norms are longitudinally valid has implications about the 
growth of Reading achievement which have not yet been independently 
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verified. For example, as iPc^asured by the MAT; Reading in children at 
the 90th percentile in the * 'ccnd grade is growing almost six times 
faster than is Reading Jr. ;iP,iiT.h grade children at the same percentile; 
Reading in second grac..^ c' lldrer at the 10th percentile is growing only 
1.5 times faster. If ..h- ♦ tr ie it is certainly an interesting fact 
for developmental psy r.hoxog.lsts to explain. On the other hand, this 
interaction of learning rate, /^rade, and percentile may just be an 
artifact of the MAT scaling process. 

Artifacts may indeed be present. In the fourth grade, where we 
did not change levels of the test, only one site out of ten gained in 
the percentile of its mean between fall and spring. At the third grade, 
eight sites of nine showed gains in percentile, as did all seven sites 
in the fifth grade. Simulations we performed confirmed these "grade 
effects" were not necessarily effects of the PIP instructional processes. 

Based on our special studies we concluded that the PIP replication 
principle was not sound: there was little reason to expect that projects 
which correctly rendered the PiPs would drc^matically increase MAT-type 
reading scores. 



Achievement lest Results 

Analysis of our MAT achievement test results confirmed the expec- 
tation that the Reading scores would not drarratically change. 

Norm referenced analyses showed that Reading scores were not 
(educationally) significantly greater than expected. The projects did 
generally maintain at least the growth predicted from the norm tables. 
However, since the growth predicted from the norm tables is generally 
an underestimate of expected growth, given that the pre- and posttests 
are not perfectly reliable, achieving more than the expectation calculated 
from the norm tables is not as impressive as it at first seems. 

Bivariate regression equations were fit to the pre- and posttest 
scores, using dummy variables to encode the effect of well-implemented 
teachers. We did not find that being in a well-implemented class was 
systematically related to gains larger than the gains in poorly- or 
moderately well-implemented classes. 
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.1 INTRODUCTION 



The evaluation of Project: Information Packages (PIFs) during the 
second year of their fiold test is described in this volume. We present 
the rationale, the data c(-^IIection and analysis procedures, and our find- 
ings with respect to instructional implementation and student achievement 
outcomes. In this introductory section, we describe the history of the 
packaging concept and each of the six PIPs created for the U.S. Office 
of Education (USOE) by RiMC Research Corporation. In addition, we describe 
the field test of the PIPs and the organizations involved. 

I . 1 Background of the PIPs 

As part of its desire to spread innovative practices from federally 
funded projects and to improve federal program performance, USOE supported 
the development of six PIPs. These packages were conceived as an alterna- 
tive to methods of dissemination that require demonstration, on-site 
training, or other hands-on technical assistance. 

The basic concept underlying the development of the PIPs was repli- 
cation. The Office of Planning, Budgeting, and Evaluation (OPBE) within 
USOE conceived of a package of instructions that would enable a school 
district to replace an ineffucclVf educational program with an educational 
program that had provud effective in another district. If such a project 
could be established at a new sir.e as specified in the instructions, the 
assumption was that the project would prove effective again in terms of 
producing equivalent student achievement gains. Presumably, the equivalent 
gains would occur because the essential program features or set of learn- 
ing conditions had been replicated by the new site. 

Another conii idera t ion in the packaging concept was that the package 
be a complete and self-contained collection of information aad instruc- 
tions assembled and structured so as to enable educators at a new site 
to select a program and implement it. The package was to contain suffi- 
cient do-it-yourself information to reproduce the program without assis- 
tance . 

In the spring of 1973, RMC Research Coiporatlon was conunissioned to 
create packages that provided "how to" information and ins truct ions to 
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facilitate the implementation in new school districts of selected com- 
pensatory reading and mathematics programs. With efforts geared to find- 
ing eight successful reading and math programs, RMC developed criteria 
for selecting exemplary programs for packaging and embarked on an elabo- 
rate search and review procedure. Over 2000 projects were suggested or 
recommended for consideration; after an initial screening, over 100 were 
subjected to a thorough review. 

Three screening criteria were used by RMC to select programs for 
packaging: (1) effectiveness, (2) cost, and (3) availability and replica- 
bility . 



Ef f ec t iveness --RMC reviewed the results of a local evaluation of 
each project to determine the validity of the data and the claims of 
effectiveness. 

The effectiveness criterion had two separate aspects, statisti- 
cal and educational significance. Both were cast in terms of 

(a) the pre- to post-test gains of project participants and 

(b) the gains which would have been expected had they not re- 
ceived the special treatment. The educational significance 
criterion which was agreed upon specified that observed gains 
had to exceed expected gains by at least one-third standard 
deviation with respect to the national norms. The statistical 
signif icance* criterion specified that the difference between 
observed and expec ted ga ins had to reach or exceed the five 
percent confidence level, using a one-tailed statistical tost 
(Tallmadge, 1974, p. iii) 

Because there were almost no cases in which a control group had been 
used in the local evaluations, PilC decided that '^expected gains" for 
students in compensatory programs 'twould mean that the group remained in 
the same percentile on both a standardized pretest and a posttest. Any 
group that exceeded these expected gains by a t least a one- third standard 
deviation (with reference to the norm distribution of the particular test) 
met RMC's criterion for educationally significant gains. For one project. 



A list of references is appended to this report, 
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Programed Tutorial Reading (PTR), RMC used the regression-discontinuity 
model to determine effectiveness. 



RMC reported that selection of programs that were successful enough 
to be recommended turned out to be extremely difficult. '*Not one of the 
several hundred project evaluations which were examined provided accept- 
able evidence regarding project impact" (Tallmadge, 1974, p. iv) . Even 
where adequate data had been collected, analysis and reporting practices 
forced inferences to be drawn with caution. To meet the effectiveness 
criterion, RMC decided that evidence of statistical and educational sig- 
nificance had to be f oun i in two instances, for example, at two different 
grade levels or for two different years. Under these conditions, only 
six projects met the effectiveness criterion and the other criteria. 

Coist- -Although attempts were made to find effective programs that 
wer^-? also inexpensive-- to make it more likely that programs would be ex- 
portable to many sites--a recurring cost of $475 per student was eventually 
established as the upper limit for projects selected, with the additional 
provision that start-up costs not exceed $1000 per student. 

A vailability and Replicabilitv --These areas were more judgmental 
and were based on the fact that the packages were likely to be dissemi- 
nated--at least on a trial basis--under the auspices of the federal gov- 
ernment. Projects were rejected on f.he criterion of availability if 
directors and ytaff were unwilling to cooperate in helping the packaging 
effort or if the project was no longer operating and could not be visited 
for validation. Projects were also rejected if they were not operating 
in public schools or if their selection would a'nuunt to a USOE endorsement 
of a single publisher's or m.Tnufac turer ' s commercial product(s). Projects 
were rejected on the criterion of repi: ca bility if they required resource*', 
not generally available in typical a .hool districts. This consideration 
included highly unusual personnel, equipment, or environments. Projects 
rejected on repl icability grounds included a university-operated elementary 
school and a project requiring major architectural modifications to the 
school building (Tallmadge, 1974, p..iii). Since USOE felt that PIPs 



RMC published a practical guide to project evaluation on the basis of 
their experiences in searching for exemplary programs. SRI assumed 
that the procedures described in this booklet by Horst, Tallmadge, and 
Wood (1975) were those RMC endorsed. These procedures form the basis 
of the norm-referenced analysis we 'used to evaluate student achievement 
in the PIP field test. 
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might eventually be disseminated under Title I auspices, RMC excluded 
from their search those programs that were not aimed at reading and 
mathematics achievement for disadvantaged children in kindergarten 
through twelfth grade. 

Although RMC had wished to find eight projects, only six satisfied 
all the criteria. RMC prepcred the program descriptions and the evidence 
of effectiveness for USOE's Dissemination Review Panel/' and all six 
projects were approved as exemplary projects. They were: 



Project Catch-Up 

Project Conquest 

High Intensity 
Tutoring (HIT) 

Intensive Reading 
Instructional Teams 
(IRIT) 

Programed Tutorial 
Reading (PTR) 

Project R-3 



Newport-Mesa Unified School District 
Newport Beach, California 

School District 189 
East St. Louis, Illinois 

School District of the City of Highland Park 
Highland Park, Michigan 

Hartford Public Schools 
Hartford , Connec ticut 

Davis County School District 
Farmington, Utah 

San Jose Unified School District 
San Jose, California 



1.2 Nature of the PIPs 

After identifying the six PIP programs, RMC developed the packages 
tlmt confainea the '''how to" information. PIPs were boxes (18 X. 14 X 12 
inches) with ten upright drawers (containing nine components) and one 
large ilat drawer at the top (see Figure 1-1). Inside the drawers were 
brochures, m/muals, Cilmstrips, cassettes, catalogues, charts, and the 
like. 



This :'.ni', ' is • w a joint activity of USOl£ and the National Institute of 
Educat-iCM I'lli.) and is called the Joint Dissemination Review Panel ( JDRP) . 
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FIGURE 1 PROGRAMED TUTORIAL READING AND PROJECT CONQUEST PIPS 
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1.2.1 PIP Components 



The first four components were concerned with planning and 
other preimplementation tasks required to equip, staff, and set 
up a new project in a school diiiitrict. They were primarily in- 
tended to assist the director in installing the adopted project. 
The remaining five components were more specifically concerned 
with project Implementation and were directly related to tne 
day-to-day operation of the project after it was installed 
(Tallmadge, 1974, p. v) . 

The components are descrihed below according to their functions and 
contents. Table 1-1, adapted from an RMC report (Plestrup, 1974), sum- 
marizes the PIP components and their purposes. 

Starter Set: PlanninR --This component provided the new project 
director with multim^idia materials to use in presenting the project and 
in gaining support of parents, teachers, principals, and school boards. 
The starter set described the features of the project for a general audi- 
ence with some elaboration on these features so that the project director 
could conduct subsequent briefings and answer questions on the project. 
This component included one-page handouts, illustrated brochures, a film- 
strip, and a tape cassette. 

Project Management Directorv --This manual brought together in calen- 
dar form the list of management activities for the project. Key tasks in 
installing and implementing the project were blocked by weeks so the 
director could fill in the actual event, such as a mefeting with the princ 
pal, on the day it would occur. The directory summarized the tasks for 
each month in a checklist to emphasize the importance of accomplishing 
the tasks as close as possible to the scheduled time. It also provided 
budget-updating summaries for each month for the director to fill in. 

Project Management Displavs --The displays were designed to attract 
attention to the existence of the project and to elicit questions or in- 
terest concerning it. The displays provided a time schedule overview 
and summarized the key tasks to be per ormed in d?:jveloping the model 
project and the components to be used in performing the tasks. 

«i 

Staff Qualif iccitions and Preparation Set --This set provided project 
directors with information on hiring personnel. It contained a staff 
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TabU- »-l 



Component 


Per Sonne I / Pur posf' 


Content s 


Installation (ittart-up/ 

I. Starter set : 
plan.iinK 


For prvijcct direi'tori 

ProviJi'j orientation to proio-t 

Provides public relations Dat«»r:al*; r>n project and 
pairkflRL* 

Provides introduction to p.ickaije (and director's rnle 
in project ) 

For s:hooI boards, principals. re<;ul.ir school staff, 
poten^'ial project statf. parents: 

Tr^/ities information to elicit support 


FilTTistrip with cossecce cape 

li.indout brochu;".' 

Project director's orienta- 
tion booklet (with cassette 
tape for some projects) 

Vieufoi Is 




Pro i tfC t nanig^ - 
neni Jirectory ^ 


For prnje^'t inanaptTnenc personnel (project director , 
«^sist,int director, principal, as applicable): 

Provides detailed guidelines and support tn.iterials 
needed to plan and implement (operate) the project 


Project management calendar 
Supplementary sections on the 
najor management tasks 




Pr o jec t "la jis^e - 
ment displo/s 


For project director .ind visitors: 
Provir'tTS Kiuie schedule overview 
i^umiPiir i ze s componen t use 

Displays component use and time schedule to visitnrs 


t^ajor management 'asks chart 
PIP use displays 




St.ll f c;'j.il ' f ic.i- 
tions and prepa- 
ration s*;t 


For director: 

Provides personnpl select ion puidelines 

For sttiff: 

Provides se 1 f -e va I uat i on and orient.ition materials 


Staff qualifications checklist 
In-service training topics 


Inplement at ion st.iiiL* 
^. Siarter $<>t: 
implementtition 


For project classroom personnel: 

Kelps in starting each type of activity (testing, 
teaching, other) including setting up environment 
for the first t ime 


Implementation starter booklet 
Or ig ^ na I art tile 


6. 


Classroom nanane- 
n>ent directorv 


For project c las sroora per Sonne I : 

Provides guidelines for classroom procedures 
Provides samples of materials needed for Jdministra- 
tion of the project, record f orms , letters and no- 
tices, and the liki.> 


Teachini; staff {;uide (e.g., a 
calendar and support materials) 


7. 


Student r I at i un- 
ships album 


Fi>r prijjert st.itf interacting with children: 

Uescribiis the project environment, from the ihild's 
viewpoint, that staff is expected to create (e.a., 
how he should perceive staff; what le.irning climate 
he should experience — high pressure, self-directed, 
and sn on) 

[»i s t in^ru i shes roles of different st.ift members in 

crii.i t irtr envi ronmen t 
npscribes desired stadenr respnnses ff.g, , confident , 

compt'tiMit , happy, csiKer) and pive-j 'Specific in?:anie« 


A I bum 


•1 . 


Prof f ss loM.i I 
re I a*, lonsh \ ps 


For all project st.iff (plus principall: 

Defines roles in rel.ition to nil schoiit st.iff (p^oj- 
*>ct n-id nofiproject) with whon i-ach r>rni^*ct member 
inter. iL't s 

Provirii*-; description!; of potential stalf conflicts ' *" 
and sncnested resolut ions 


Guide booklet 




fi.ir 1 - 1 1 ■■ s il tw.irt" 


For projeft iirnctnr .ind teacnint* st.itf: 

Aids in so 1 i.'c 1 1 on and ordering of t-ont-i-rc i a 1 h ard - 

w.i r.'/ T-o f t u.irt* 
Prov I des description, snur ce , and f t-.it ure s o t cnre 
find supp I entMit ii ry i t ems 

iFnr ti'.ichmr statt: 

[ r.escribes experience n { orijiin.il proje. l itaff plus 
nod i : i c a 1 1 on f i f .iny / 


Factsheets 3,id manufacturers' 
brochures for core i terns 

Supplementary materials list 
with publishers* riddresses, 
available factsheets, and 
brochures 


S.uirv; 
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qualifications checklist, information on training topics, and a suggested 
agenda for training project teachers and aides. 



Starter Set: Implementa tion --This component provided information 
for the early weeks of instruction in the new project and included ma- 
terials for decorating the classroom, a detailed calendar for the first 
two weeks of school, and descriptions of how to begin each new activity 
(testing, teaching, and the like). This starter set enabled new staff 
to create an attractive environment appropriate for the region where the 
project was to be implemented. 

Classroom Management Directory --This directory, designed to corre- 
spond to the Project Management Directory in format and purpose, contained 
day-by-day management instructions for the classroom teacher. Calendar en- 
tries indicated the sequence of events and reminded teachers to perform key 
tasks throughout the year. The directory included monthly task summaries 
in a checklist, budget records, and supplementary sections explaining 
alternative strategies for accomplishing tasks and anticipating problems 
and described the practical details of operating an instructional system. 

Student Relationships Album — This album described the roles of staff 
members in relation to students. Role illustrations were presented to 
convey the **flavor" of the project fiom^a child's viewpoint. The album 
dealt with the difficult areas of attitude change, self-concept develop- 
ment, fostering achievement orientation, and eliminating prejudice. 

Professional Relationships Guide --This guide described problems and 
conflicts that occurred at the exemplary site and suggested ways of form- 
ing good working relationships in the new project. The guide defined the - 
roles and responsibilities of project and nonproject staff in relation to 
one another. 

Hardware/So ftv;a re Packet --This packet provided information on in- 
structional macerials and equipment, some of which were optional. The 
packet contained publishers* and manufacturers* brochures describing the 
materials and instructional hardware to be used. 

Flat Art File --This file contained reproductions of display materials 
developed by the original project and instructions for the production of 
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other materials. It enabled the adopting school to make use of project- 
developed materials designed to enhance motivation, to improve self- 
concepts, or to facilitate skill development. 

In creating the content materials for the components of the PIPs, 
HMC assumed the following principles: 

• It was necessary to match the setting of the replicating 
site with the setting of. the original site. An Analysis 
and Selection Kit (ASK) was designed to provide local edu- 
cacion agency (LEA) staff with information by which to 
choose the exemplary program best suited to their needs 
and most likely to be fully implemented in the context of 
their school district. (See Section 1.2.2 for a descrip- 
tion of the ASK.) 

• The project director at the new site was the critical 
element, and management of the installation process was 
the key to replication of the original program in the new 
site. Information provided in the PIPs was described to 
enable project directors to set up the conditions for the 
effective instructional program to be recreated. 

• Project directors were to select instructional staff who 
were exemplary teachers in the same mold as those in the 
original project. It was assumed that teachers could not 
be trained to have certain instructional theories and 
methods but that they could be selected to fit the project 
mo d e I . 

• Minimum information to guide staff training and to guide 
instructional practices would be sufficient if staff se- 
lection were appropriate. 

• A list of curriculum materials used at the original site 
was sufficient to permit replication of the originally 
effective curriculum and instructional program, since 
qualified staff would choose appropriately among the ma- 
terials to serve the individual needs of the students. 

1.2.2 Analysis and Selection Kit 

Because the introduction of the packages to potential users and the 
processes within the LEAs for selecting and adopting a program were not 
encompassed by the PIPs, an Analysis and Selection Kit (ASK) was designed 
The prototype kit consisted of the following: 
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• A brochuro for wide disscmina cion, describing the six 
exemplary programs and their availability in the form of 
PIPs. (This kind of brochure had been available for other 
package dissemination programs such as that sponsored by 
the Right- to-Read program.) 

• Further information, available upon request, about each 
program. (This information was to enable an LEA to choose 
the program best suited to its needs.) 

• A set of orientation materials (actually the very materials 
contained in the "Starter Set:Planning" component of the 
PIP) to enable LEA staff to orient school board, community, 
instructional staff, and the like to the selected program. 

The ASK had barely re'ached the prototype stage when USOE found an oppor- 
tunity to test the PIPs in the field. 

1 . 3 Description of the Six Original Programs 

The six exemplary programs, as they existed at the time they were 
examined by RMC, are described below. Table 1-2 summarizes the character 
istics of each. 

1.3.1 Project Catch-Up 

Catch-Up in the Newport-Mesa Unified School District in California 
provided remedial instruction in reading and mathematics to low-achieving 
children in a low-socioeconomic suburban area. The program served chil- 
dren from preschool through twelfth grade. (The Catch-Up PIP described 
only the elementary portion of the program, kindergarten through sixth 
grade.) The major instructional emphasis of the exemplary program was 
on diagnosis of learning problems through extensive use of criterion- 
referenced materials. A special staff of certified teachers diagnosed 
student problems and prescribed activities and materials for teaching 
two or three specific skills. The learning experience was provided by 
certified part-time teachers and instructional aides in an attractive 
**labora tory" environment. This lab was set up within an available class- 
room in the school and operated only in the morning when both staff and 
students were fresh. Students attended the. lab 20 to 60 minutes daily, 
depending on their need for extra reading and mathematics instruction. 

It was essential to the philosophy of the program that each teacher 
be responsible for the achievement gains of no more than 18 students. 
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CllARACTERtSTlCS Of THE SU mWim PIP PROCR,\MS 



Program 


Catch-Up 
Newport Beach, Californii 


Conquest 
East St. Louis, Illinois 


High intensity Tutoring (HIT) 
Highland Park, Hichigan 


Subject areas 


ReadinR a»d math 


Heading 


Reading and math 


Students 
stirved 


Students in grades H who need help 
in reading or math; average student- 
teacher ratio, 5:1. 


Students in grades 2-5 who arc two 
years or less below grade level, and 
first grade repeaters; student-teacher 
raiio, o>Li 


Students in grades 6 or ? one to five 
years below grade level tutored by stu- 
dents in grades 1 or B, at least two 
vpar^ fthpflH of them on a onp*'to"Onp 

basis. 


him of 
program 


Students come from classrooms to the 
Catch-Up lab every morninf;. Ttio lead 
teacher, the two other teachers, and 
•the aide work independently with chil- 
dren for W minutes. Pai'ent aides as- 
sist in instruction. 


jiUQcncs come ironi cias^iii^oms rt^tia 
ing roocis (grades 1-3) or clinics 
(grades ^-i) for W or 50 minutes each 
day. Heavy diagnostic testing Jeter- 
mine, areas that the lab clinicians 
will work on with each student. 


flnp riitftr uork^ with nne tutee usinc oro- 
grammed curriculum for 20 minutes under 
supervision of a reading or math special- 
ist and two aides. Rewards are earned 
by tutors as well as tutees. 


Pto^raiA 


Intensive Rijading 
Instnictionjl Teams (IRIT) 
Hartford^ Connecticut 


Propraracd Tutorial Beading (PTR) 
FarmingLon, Utah 


Project R-3 
San Jose, California 


Subject areas 


Reading 


Reading 


Reading, math, and social studies 


Students 

9t. I VtlU 


Students in ^ndcs 3 and ^ who need 
hf»!n in jir<M^ of lancuacti arts I 
Student-teacher ratio, 15:1, 


First graders who need help learning 
to read are tutored on a one-t.i-one 
basiSi 


In the project's first year, all seventh 
graders in heterogeneous classes (gifted 
and slow learners); studont-teacher ra- 
tio, 20:1. 


iNaturf! of 
prograni 


Forty-five students from several 
schools are brouf;ht to one site each 
morning for 11 weeks for intensive in- 
struction by three language arts spe- 
cialists in three onc-hour sessions. 


Students are tutored apart from their 
regular classrooms for 15 minutes a 
day by paraprofessional tutors; pro- 
grammed kits designed for the basal 
reader used in the classroom are the 
only curriculum materials uaedi 


Forty-five minute periods of reading, 
math, and social studies offer an inte- 
grated curriculum orientod to gaming/ 
siiitulationi individualized instruction, 
and an intensive involvement study trip. 
A cadre staff (oife teaclicr per subject) 
moves with students to grades 8 and 9i 



1.3.2 Project Conquest 



Conquest in East St. Louis, Illinois, used a clinical approach to 
reading and served students in grades K-6 who were two years or less be- 
low grade level. The instructional staff diagnosed each child's reading 
problems through an intensive 17-step diagnostic procedure and prescribed 
a structured learning program to be followed by each child. Remedial 
instruction was provided in 45 -minute sessions held four to five days 
per week. The teachers, called clinicians, received extensive in-service 
training and supervision in remediation techniques, testing, diagnosis, 
and related areas. The centers were designated as clinics (for grades 
4-6) or reading rooms (for grades K-3) and were separate from the regular 
classroom, serving several groups of students per day. 

The Conquest student expejtie.nced three or four activities in the 
following areas during each session: programmed reading, comprehension, 
phonics/ vocabulary/ sight words, and oral or recreational reading. In- 
struction in at least one of the areas was assisted through the use of 
a teaching machine. 

1.3.3 High Intensity Tutoring (HIT) 

HIT in Highland Park, Michigan, was a peer tutoring program that 
used highly structured materials for sixth, seventh, and eighth graders-. 
Older students tutored younger ones daily in reading or mathematics, or 
both, using programmed and drill materials. Both tutors and tutees were 
performing one to three years below grade level on standardized tests. 
Tutoring was fast-paced and intense. The percentage of correct responses 
for each tutee in the program was calculated daily to ensure that presen- 
tation of new materials was adjusted to the student's rate of learning; 
the goal was for students to achieve a correct-response rate of 907o-947o 
each day. Interaction between tutor and tutee was structured to maximize 
the time that each tutee was engaged in active learning behaviors. Tutors 
checked each answer as it was made and provided correct answers and rein- 
forcement according to a structured procedure. Rewards were an iucvative 
device for both tutors and tutees. A unique feature of the HIT program 
was that the tutor also improved academically during the learning process. 

There were two HIT centers at each school (a reading center and a 
mathematics center). Each waf staffed by a certified teacher and two 
paraprof essional aides. To control the student error rate, teachers and 
aides monitored the tutoring, distributed rewards, and kept detailed 
records. Teachers administered Sullivan pretests to establish an entry 
level into the materials for individual students. 
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1.3.4 InCensive Reading Instructional Teams (IRIT) 



The IRIT program In Hartford, Connecticut, was designed to raise 
the level of language and readlrg achievement of third and fourth grade 
pupils who were deficient In the basic skills of language and reading. 
IRIT employed a team- teaching approach with three speclalls ts--f or phonics, 
Individualized reading appreciation, and vocabulary/comprehension. Diag- 
nostic testing identified the special needs of each student. The indi- 
vidualized reading sessions offered reading assignments that enriched 
the child's background, promoted his written and oral language skills, 
and instilled pleasure in reading. The vocabulary and comprehension 
sessions built perceptual and reading skills. 

The IRIT program was divided into three Il-week cycles. Forty-five 
students were selected for each of the three cycles. During the cycle 
assigned to them, students left f.beir regular classrooms to go to the 
IRIT classes for three hours each morning, five days a week. The 45 
students in the program were hef: erogeneously divided into three groups 
of 15 each. During the three-hour morning, each group of 15 students 
spent one hour on phonics, one on individual reading, and one on vocabu- 
lary and comprehension. 

1.3.5 Programed Tutorial Reading (PTR) 

PTR* in Farmington, Utah, provided tutoring to underachieving first 
graders in beginning reading as a supplement to conventional classroom 
teaching. The tutoring materials included the same basal readers that 
were used in the regular classrooms, along with a comprehension and word 
analysis book and word list cards. 

The teaching strategy employed many of the elements of programmed 
Instruction: frequent and Immediate feedback, specified format, and 
individualized pace. However, whereas programmed instruction had often 
sought errorless or nearly errorless learning with many cues at first, 
followed by a fading of cues, the PTR proceeded in the opposite manner 
with minimal cuing at first, followed by Increased prompting until the 
child could make the correct responses. 

First grade students were tutored on a one-to-one basis by care- 
fully trained tutors for 15 minutes each day. The tutors were parapro- 
fesslonals who ranged in skill and experience from high school students 
to teacher aides and community members or paronts. 

The PTR program was originally created by Douglas Alison at the 
University of Indiana. 
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1.3.6 Project R-3 



*'R-3" stajids for student readiness , subject relevance , and learning 
reinforcement . The original program in San Jose, California, was a junior 
high school program for reading, mathematics, and social studies in which 
the teachers engaged in team planning to create a highly motivating pro- 
gram for the students. All seventh grade students participated in R-3; 
the program followed these students as they moved on into the eighth 
grade, and then into the ninth grade. Each student attended one 45- 
minute period in each of the three subject areas daily. Because R-3 
served an entire grade, the group was heterogeneous (i.e., not composed 
only of low achievers). 

Important components of the R-3 program were gaming/simulation ac- 
tivities, learning contracts^ individualized instruction, intensive in- 
volvement study trips, and parent involvement. Gaming/simulation rein- 
forced skills learned in each subject area. Contracts encouraged the 
student to set his own goals and to work independently; each student was 
held responsible for completing his contract. Teachers used an eclectic 
approach to instruction and to their use of instructional games, simula- 
tion, contracts, and intensive involvement. Project staff naade home 
visits to involve t. e parents in the child *s program. 

1.3.7 Summary 

The six exemplary programs were different from one another. Sonie 
were for elementary grades, others for middle-school. Some required non- 
professionals as instructors; others required highly qualified reading 
specialists . 

Although each program was unique, certain features were similar. For 
example, two of the six progra:;ns, PTK and HIT, were tutorial programs in 
which instruction was provided on a one-to-one basis. This allov;ed im- 
mediate feedbacks a specified format, and an individualized pace. The 
few curriculum materials used were programmed to maximize the amount of 
time the tutee was engaged in active learning. Both projects employed 
supervised students or paraprof essiona 1 staff to monitor the tutoring 
process. Three programs, IRIT, Catch-Up, and Conquest, were laboratory 
programs. They all used a diagnostic-prescriptive instructional approach 
in which learning deficiencies were identified and a curriculum prescribed 
for each student according to hi£ needs. Students in these programs spent 
more time receiving treatment than did students in the tutoring programs. 
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All except Project R-3 were '*pull-out'* programs. That is, they sup- 
plemented rather than replaced regular classroom teaching; they required 
students to leave class and go to another location to participate. In 
addition, except for these p7*ograms did not require regular class- 

room teachers to make changes in their methods or behavior other than 
adjusting schedules so that children could be released to participate in 
the project. 

In R-3, all seventh grade students participated and continued in 
the program during the eighth and ninth grades. Unlike the other projects 
which emphasized reading or reading and math, R-3 included three subject 
areas: reading, mathematics, and social studies. 

1.4 Field Test of the PIPs 

In February 1974, while RMC vas developing the PIPs, the Title III 
program office announced the availability of four types of grants under 
Section 306 and sent application requirements to all schooT districts. 
The announcement of Title III grants contained no information about the 
six PIP programs except that they were reading and mathematics programs 
for disadvantaged elementary and secondary students. Although E^C had 
sent OPBE some of the information LEAs would need for selecting a program 
suited to their needs, they had not yet put infoiination about all of the 
six PIP programs into a brochure. 

After a joint review of applications by Title III program officers, 
OPBE staff, and consultants, 21 PIP grants were awarded. One school 
district, which received grants for both the PTR and the HIT PIPs, re- 
turned the award after its board of education refused to approve partici- 
pation in the program. Since two districts had been assigned two PIPs 
eazh^ the final tally was 19 projects in 17 school districts. Table 1-3 
lists the school districts participating in the field test of the PIPs. 

The field test of the six PIPs was an opportunity for OPBE to try 
out the PIPs in school districts and to evaluate the feasibility of the 
PIPs as a method by which USOE could disseminate programs that would im- 
prove the reading and mathematical skills of disadvantaged children. 
OPBE called for a '^comprehensive portrayal and analysis of the process'* 
from which to decide whether to continue, terminate, or modify the six 
PIPs and the packaging concept. 

The principal criterion by which USOE intended zo judge the effective 
ness of PIPs wa^^ that of educationally significant growth in achievement. 
From USOE's point of view, if the student achievement test outcomes in the 
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Table 1-3 



SCHOOL DISTRICTS PARTICIPATING IN PIP EVALUATION 



PIP 


Proiect Location 


School District 


Catch-Up 


Blooming ton, Indiana 
Brookpor t, Illinois 
Galax, Virginia 
Providence Forge, Virg inia 
Wayne City, Illinois 


Monroe County Community Schools 
Brookport School District 
Galax City Public Schools 
N ew is.en c uo un. cy ruDiic oc noo is 
District No. iOO 


Conquest 


Benton Harbor, Michigan 
Lieveiana, unio 
Gloversville, New York 


Benton Harbor Area Schools 
Lii.evei.anQ i:UDi.i.v. iiJv-iiuui.o 
Gloversville Enlarged School District 


HIT 


Lexii.':*'On, Mississippi 
Olean, New York 


Holmes County School D is trie t 
City School District of Olean 


IRIT 


Bloomington, Ind iana 
Oklahoma City, Oklahoma 
Schenectady, New York 


Monroe County Community Schools 
Oklahoma City Public Schools 
Schenectady City School District 


PTR 


Canton, Mississippi 
Dallas, Texas 


Canton Public Schools 

Dallas Independent School District 


R-3 


Charlotte, North Carolina 
Lake Village, Arkansas 
Lorain, Ohio 
Schenectady, New York 


Charlotte-Mecklenburg Schools 
Lakeside Public Schools 
Lorain City Schools 
Schenectady City School District 



field-test sites showed gains equal to those of the original programs, 
the PIPs could be judged successful in terms of program effectiveness. 
For effectiveness to be determined, pre- and post-tests of participating 
students were required. USOE specified the following criteria for se- 
lecting test batteries: 

The belected battery is to include subtests, at all grade lev- 
els, to assess vocabulary, reading comprehension, word analy- 
sis skLj.Is, mathematics concepts^, reasoning and computation 
skills. The selected battery (or batteries) must further meet 
the following criteria: \ 

1. Provide evidence of subtest reliability and validity. 

2. Contain interlocked forms for grades kindergarten through 
12 (K-12). 
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3. Contain alternate forms for pre- and post- testing. 

4. Be basic-skills oriented (i.e., subtest items should be 
independent of specific curriculum content). 

5. Be of an appropriate difficulty level for the student 
sample . 

6. Include in the norming sample students similar to those 
in the treatment and comparison groups of the field test. 

7. Be easily administered and scored. 

8. Require a reasonable amount of administration time (HEW 
Request for Proposal 74-40, p. 18). 

In the test selection task, it seemed appropriate to consider the 
seven standardized achievement tests included in the Anchor Test Study. 
These tests were: the California Achievement Test, the Cooperative Test 
of Basic Skills, the Iowa Test of Basic Skills, the Metropolitan Achieve- 
ment Tert, the Sequential Tests of Educational Progress, the SRA Achieve- 
ment Tests, and the Stanford Reading Tests. 

Of these instruments, the Metropolitan Achievement Test (MAT) was 
the only one for which both fall and spring normative data were available. 
Since it met more of the criteria than did the other tests, it was sched- 
uled for the PIP field-test evaluation. Discussions with personnel at 
the UCLA Center for the Study of Evaluation revealed that the 1970 edi- 
tion of the MAT was among the highest rated tests in the CSE Elementary 
School Test Evaluation report (Hocpfner et al., 1970). 

1 .5 Organization of the Evaluation 

During the past two years many groups were involved in the evalua- 
Lion study. As mentioned earlier, the Office of Planning, Budgeting, 
and Evaluation within USOE initiated the packaging concept. The Division 
of Centers and Supplementary Services in USOE's Bureau of School Systems 
awarded grants under ESEA Title III, Section 306, to 17 school districts 
to implement the PIP projects. 

Each of the 19 tryout projects had an organizational configuration 
based on the instructions in its PIP. Each of the six PIPs called for a 



The Anchor Test Stiudy (Loret et al. , 1974) was the result of USOE ' s artempt: 
to devise an approach for equating scores f^om different reading tests. 
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project director to manage the project, to hire and train the instruc- 
tional staff, and to orient other disiii'ict personnel and the public to 
the PIP program. 

USOE awarded a contract to SRI to evaluate the PIPs in their two- 
year field test. SRI in turn subcontracf~ed with RMC for the formative 
evaluation and a revision of the six RMC-developed PIPs. During the 
first year of the study, ?HC staff accompanied SRI staff on field visits. 
During the second year, RMC made revisions based on the findings of the 
first-year evaluation and continuing input from SRI staff members as the 
main evaluation progressed. The second-gener -j tion PIPs represent attempts 
to fill the gaps and to resolve the ambiguities identified during the 
evaluation of the first-generation PIPs. 

To foster understanding of their education programs, directors of 
the exemplary projects previewed assistance uo both SRI and RMC. The 
exemplary projects also provided some minimal assistance, upon request, 
to the PIP field tryout sites. 

The SRI evaluation focused on the degree of imp 1 omen ta t ion of the 
PIP-specif ied installation activities at each site during the first year 
and on the effectiveness of the projects in terms of student achievement 
during the second year. In both years the evaluation plan called for 
extensive fieldwork through observations, interviews, student testing, 
and collection of information regarding curriculum materials. Distinct 
groups performed the functions required for -nch of these data collection 
activities. Site visitors were the ba^ic da lj collection unit. Each of 
these six professionals was expert in a t least one PIP program. Their 
function was to visit their assigned project sites several times a year 
to observe the PIP implementation process and to interview the educators 
on site. It is upon their judgment that our findlni^o about degrees* of 
implementation depended. 

Althougli student achievement was the primary focus of the second- 
year evaluation, achievement and attitude tests were given to students 
during both years. SRI used its own personnel to administer the pre- 
and post-tests in fall and spring. At sites having a large number of 
students to be tested, SKI administrators hired and trained qualifl':'d 
personnel at the test site to give and monitor the tests according to 
SRI procedures. 

Because SRI personnel arc stationed in Menlo Park, California, site 
assistants were hired at. each of the project sites to help the SRI evalu- 
ation staff. These local site assistants were hired by the site visitor 
upon recommendations hiade by the local project director. The principal 
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duties of the site assistant were to set up the logistics of the site 
visits and to perform the clerical tasks associated with the testing of 
project students (e.g., scheduling testing, identifying students to be 
tested, labeling test booklets, and shipping test materials back to SRI). 
In addition, when needed, the site assistant provided SRI with copies of 
local documents, such as student lesson plans and lists of curriculum 
materials used by the PIP instructional staff. 

1 - 6 Evaluation Issues 

The central evaluation issue for the second year report is the 
validity of the replication principle. This concept is that the PIPs 
would induce projects which raise achievement test scores to the same 
degree as the original projects, if the original project and PIP project 
were given the same norm-referenced achievement test. 

This idea of replication is rooted in some widely held beliefs about 
the worth and equivalence of some nationally normed achievement tests. 
First it is believed that raising scores on these tests is desirable and, 
secondly, it is believed that, at least among the tests used in the Anchor 
Test Study (Loret et al. 1974), most achievement tests are equivalent. 
Thus, the replication principle does not assert that PIP effects are test 
specific. Indeed, many policy makers would argue that projects with such 
specific effects would not be worth disseminating. 

Our first year results, which showed definitely unspectacular test 
gains, led us to question the validity of the replication principle. 
However, we could not definitely assert that v;e would not see the expected 
results for this report. 

Consequently, we decided to execute three types of analyses. The 
first analysis used the concepts underlying the replication principle. 
These results are reported in Section 3-7. However, based on our first 
year's experience we knew that the analysis techniques used in Section 3.7 
were flawed* The full extent to which they were flawed and the importance 
of the flaws were not clear. 'Jherefore we undertook a second type of 
analysis, called a special study. The special studies examined, both 
analytically and empirically, some of the assumptions underlying the 
replication principle. The analytic results are discussed in Section 2.3 
and the empirical results ::re in Section 3.8. We call the third type of 
analysis we did a curriculum-referenced analysis.. 

The curriculum-referenced analysis is described in Sections 4, 5, 
and 6. This analysis was based on the premise that test scores go up in 
response to a well-implemented, appropriate presentation of a curriculum 
which is relevant to the test iiems . In our interpretation of this 
concept, the curriculum must, in the judgment of a competent specialist, 
appear relevant to the ti?st items. We assumed that the PIP-spec if ied 
teaching scyle had already been shown to be relevant. Presumably if 
PIPs worked, tho wo. 1 1- i.mplemented teacher would show better gains than 
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poorly-implemented ones, if the MAT was equally suited to the PIP-specif ied 
(and used) curriculum. The extent to which we were able to show this 
effect is discussed in Section 6. 



To summarize, the main evaluation issues were: 

• The validity of the replication principle and the associated 
norm-referenced analyses » 

• The results of the norm-referenced analyses: Did the 
replication principle predict our test results? 

• The type and impact of formal errors in the norm-referenced 
analysis . 

• The results of the curriculum-referenced analysis. 

The curriculum-referenced analysis involved several furthar issues: 

• What Was the PIP-specif ied instructional style? 

• To what degree was this style implemented by teachers? 

• What was the PIP-specif ied curriculum? 

m To what extant was this curriculum used? 

• Is the Metropolitan Achievement Test sensitive to the 
curriculum which was specified and used? 

• Did teachers who were well-implemented show better gains 
on the MAT (given the age, race, and sex distributions 
of their classes) than poorly- or so-so-implemented 
teachers? 

The next section discusses some of the analytic foundations of the 
replication principle and the associated norm-referenced analyses. 
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2 STRATEGY OF THE SECOND-YEAR EVALUATION 



2 . 1 Introduction 

The Analysis and Selection Kits (ASKs) and the Project Information 
Packages (PIPs) under scrutiny in the SRI field evaluation were thought 
of as examples of "strong packaging." That is, they were prescriptive 
instructions for selecting and implementing specific educational projects. 
Each set of instructions was intended to be, by itself, sufficient for 
implementing a copy of a previously successful project, provided of course 
that the project personnel were willing to make a good-faith effort to 
follow the ASK-PIP instructions. \ 

The fundamental motivating principle behind PIP packaging was faith 
in the "black-box" concept of scientific investigation. This familiar 
research model was adopted from engineering problems in which a closed 
system, the black box, is assessed by measuring its inputs and outputs. 
The events inside the system are not necessarily important for predicting 
outcomes; it is necessary only to relate observable inputs to observable 
outputs . 

The design of the PIPs applied this input-process-output concept at 
several levels of detail. At the grossest level, when the ASK instruc- 
tions were followed, the PIPs were to be put into a socioeconomic context 
that matched the context of the originating site. The PIP itself was to 
supply the "process" of education. Educationally significantly improved 
achievement test socres were to be the expected output. At a finer level 
of detail, the PIP design told the would-be implementer how to hire 
qualified staff, how to order materials, and how to manage relations 
between project and nonproject staff and betv -jn project staff and 
students* parents. These events were the input to the black box that 
was thought of as representing the educational process. The output of 
the process was to be the already-mentioned educationally significant 
gain in children's test scores. 

The ASK-PIP combination, then, was designed to replicate an originating 
project by duplicating the project's obvious inputs: socioeconomic context, 
staff, community relations, and materials. The educational process in the 
originating site was, by and large, thought of as a black box. As a 
consequence, the PIPs relied on projects having hired teachers who already 
knew thesG details. The PIP-:, reliod on having the correct input; the 
the output was assumed to follow. 
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This volume of the report on the results of the PIP field trials will . 
focus on how the black-box approach to strong packaging has fared. In 
particular, we will examine the claim that the PIPs would produce projects 
that would cause students to achieve the educationally significant growth 
that the original projects did. 

Our measure of growth was based on the 1970 Metropolitan Achievement 
Test (MAT). The obvious first step for detecting educationally signifi- 
cant growth on this or any other test was to define "educationally signifi- 
cant growth" so that it could be identified as an output of the originating 
project; it would then be possible to detect the same event on the output 
of the replicating project. 

As discussed in Volume One, the originating projects were selected 
in part on the basis of their standardized achievement test outcomes. 
Consequently, the process for selecting the originating sites defined 
the sense of "educationally significant growth" that was appropriate to 
this evaluation. 

The selection process, as described in the Dissemination and Review 
Panel literature, called for the analysts to prefer projects that could 
demonstrate that students in the project had an average growth of one- 
third of the norm group standard deviation over that growth expected with- 
out the project. Projects Catch-Up, HIT, IRIT, and R-3 were chosen in 
this way. PTR was chosen on the basis of treatment-control group achieve- 
ment test differences. 

All projects were originally evaluated using different s tandard if.ed 
achievement tests, except for Conquest, which was selected without 
reference to any normed test. 

The black-box model predicted that the replicating projects would 
achieve the same criterion (relative to educationally significant growth) 
that the originating sites achieved. This criterion however, was not a 
function of the project only; it was also a function of the measurement 
methods associated with the several normed tests used in the selection 
of originating projects. This is unfortunate because the theory and the 
application of nationally normed achievement tests are not very satis- 
factory. It became necessary to perform the evaluation while dealing 
with technical problems not necessarily relevant to the central issue: 
Are PIPs examples of a dissemination strategy that is worthy of continued 
federal support? 

To remain focused on this central issue, we desip.ned our evaluation in 
two segments. The first segiT'enr follows the spirit of tlie black-box model, 
accepting the mca su reraen r theory v;hich It assumes when applied to the TIP 
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projects. The second segment reanalyzes the test data from a point 6f 
view that is not as dependent on the measurement theory. When data are 
examined from two points of view, the conclusions they imply may differ. 
The PIPs may be successful if judged by the black-box approach and not 
successful if the associated measurement theory is rejected. We, as 
evaluators, have an obligation to state wViich analysis we credit. We 
feel that the best analysis rejects the measurement theory implicit in 
the black-box outcomes. In section 2.2, we describe our two analysis 
designs and explain our preference. 

2. 2 Justification for Norm^Referenced Analysis 

Insofar as achievement test scores are concerned, the rational 
selection of projects suitable for strong packaging was accomplished by 
assuming what is equivalent to "generic true-score test theory," as 
discussed by Lord and Novick (1968, p. ]64). (Nontest criteria for the. 
selection of projects are described in Volume One. ) 

The theory of generic true scores treats observed test scores as if 
they were based on the following sampling scheme; 

• Randomly select n persons from the infinite population of 
humans . 

• Randomly select N tests from the infinite population of 
achievement tests; administer them K times to each person. 

- • Assume that the K y N test scores on these tests are 

independent for a given individual. 

We may then write the following standard random effects analysis of 
variance model for an observed test score, Y j . Let ^jY^^ be the expec- 
tation of Y with respect to the population indexed by j; then the model 
is : 

Ygak = ^ ^a"^ ^^^g ^S^^ ' ^^"^^ 

where 

g indexes tests g = 1, G 

a indexes individuals a = 1, A 
k indexes replications k ^ 1, K 
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^gak the specific error of mecisuremen c ; Ag is the effect of the ath 
individual; Pg is the effect of the gth test; AP^g is the interaction of 
thtj ath individual and the gth test. 



Using this conceptualization, we may talk about the various tests 
used to assess the outcomes of the originating projects as being parallel 
achievement test forms." In practice, of course, the forms are only 
nominally parallel (see Lord and Novick, 1968, p. 174). Obviously, the 
choice of a particular test is not important, as long as it is a member 
of the universe of acceptable tests. As discussed in Section 3, our 
choice--the MAT--is an acceptable measure of outcomes in the sense of 
generic test theory. 

The user of the model (Eq. 2-1) conceives of estimating the generic 
true score. T^, defined as the expectation of ^gg^\^ over the hypothetical 
universe of tests and replications. In this sense, for a given individ- 
ual, all the sampled achievement test scores in the universe estimate 
the same thing- -not the same specific true score, Ej^Yg^i^, but the same 
ge neric true score, ^g^'-K^^g^.^k • possibje to use the random effects 

model (Eq . 2-1) Lc' estimate generic true scores and their variances if 
there is a data set in which persons were given multiple tests. Unfor- 
tunately, no such data were available on the set of potential originating 
projects, so the generic true-score model was not used in the selection 
process, except to justify treating different normed tests as generically 
ucjuiva le.it . 

B.jsed on the idea of generic equivalence, if only one test were 
available from the universe of tests, the best estimate of the generic 
true score for i j.roup of project children is that group's average of 
observed scorc^j. Our best method of comparing programs is on these av- 



The Anchor Test St udy ( 1974) applies these idr^as t • the Mi\T and 7 other 
tests. The obvious criticism of the Anchor Test Study and the PIP 
replication principle is that they both ignore differences in item content. 
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erages. A reasonable plan for selecting projects that increase generic 
test scores would be to select projects that moved the average observed 
test scores higher than would be expected without the project. 

f.. s.-f:-,' 

The two questions that immediately arise are how much more than ex- 
pected^ and how much is expected. The first question was settled^ somewhat 
arbitrarily^ by accepting projects that had a growth of one-third of the 
norm group standard deviation over expected growth. The question of how 
to define **expected growth** was answered by using a control group average^ 
if one was available. If one was not, the test's norm tables were used 
to estimate expected growth^ as follows: 

• Compu te the average test score in. the fall . Cal 1 this F . 

• Find the fall percentile corresponding to F. Call the 
fall percentile f'. 

• Find f' in the spring percentile tables. Let S be the 
corresponding spring score. S is taken as the expected 
spring average score. 

Evidently^ there is a ^'sampling error'* involved in F^ and the 
observed spring score^ 0. This error should be taken into account by 
making justifiable assumptions about the joint distribution of F and 0^ 
as estimators of average generic true scores^ and then deducing what a 
reasonable statistical test might be. However^ the procedure actually 
followed was to assume that 

%-l r o ' 

+ . 2rxySxSy 

V N^"l 

is approximately distributed as student's t with N-1 degrees of freedom, 

observed average spring test score 
expected average spring test score 

observed pretest standard deviation ' 
observed posttest standard deviation 
observed pre-pojit correlation 
number «'f rhilcreri v;iLh pre- and post-scores. 

Calculation oC this statistic is Ci^Iled a "norm-referenced analysis," 
after Horst, Tallmadge, and Wood (1975). 
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The parts of the evaluation that are true to the design principles 
of the PIPs are based on the conceptual measurement associated with Eq . 
2-1. The ability of the replicating projects to move generic test scores 
is assessed by computing the norm-referenced analysis based on the value 
of standard scores obtained from the 197G version of the MAT. If the 
lower limit of the 957o confidence interval^, based on the t test (Eq. 2-1) 
for the difference between observed and expected scores^ exceeds one- 
third the norm grou^ standard deviation^ we can say that the criterion 
has been achieved. 



2 . 3 Special Analytic Studies of zhe Norm-Referenced Analysis 

As stated^ we feel the norm-referenced analysis should not be taken 
as the principal test of the validity of the PIP concept. The measure- 
ment framework iir.posed by the adoption of generic test theory^ and the 
computation of the norm-referenced analysis^ should be viewed as devices 
for expediting the selection of projects for packaging. These devices 
should be viewed^ and were viewed^ as rules of thumb, not as serious 
models for selecting exemplary programs. The next paragraphs provide 
our justification for these remarks. 

The generic true score may well be a useful construct for situations 
in which tests are actually sampled and in which there are replications 
of tests and samples of persons who have taken them. Certainly^ the cor- 
resp?nding random effects analysis of variance has found useful applica- 
tions. However, there was nothing in the selection of the PIP originating 
projects to justify the introduction of generic true scores; there was 
only " conceptual" sampling. Because generally orl}^ one test was associ- 
ated with each originating project^ it is impossible to compare the orig- 
inating proiects on estimates of their generic scores. 

As a result; we are forced to acknowledge that the PIP projects were 
selected on the basis of several specific scores^ no matter what the con- 
ceptualized sampling may have been. Similarly^ we muse view the Mi\T 
achievement scores as specific scores because we cannot compute any of the 
parameters associated with the generic scores, since the MAT was the only 
test .^iv^Mi. The results of the nonn-- referenced procedure are not opera- 
Lioral xy ccviV^ le the /i:Iiuation of the originating project although, 
in the sense of generic true -score theory, they are (conceptually) esti- 
mating tlie same thing. 

In short, the norm-r ^ t v^renced analysis of the MAT achievement data 
is not operationally ider.cical to the "educationally significant" output 
that was found at the originating sites. We do not know whether the orig- 
inating site won^d have pa«=sed the l^iAT-based norm-re f erenced test of edu- 
Cri t iona I s ii^ii .i i i c :\ ncc . 
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Additional reasons for not regarding the MAT norm-referenced analysis 
as critical stem from the properties of the MAT standardization and from 
the t test (Eq. 2-2). 



2.3.1 Criticisms of the MAT Standardizing Procedure 

A norm-referenced procedure is only as valid as the norms on which 
it is based. The 1970 Metropolitan Achievement Test, published by 
Harcourt Brace Jovanovich, Inc., was made to 1970 test industry standards. 
The test is in some ways superior to other tests of that period, princi- 
pally because the MAT has empirical norms for both fall and spring." 

The test makers provide a table of standard scores that are supposed 
to enable users of the MAT to test out of level and to measure directly 
the "amount of achievement" on an interval scale. The publishers state: 

The standard score scale for the Metropolitan Achievement 
Tests provides two basic conveniences for the test user. 
The scale makes forms within a battery equivalent and pro- 
vides a continuous, equal interval system for each test 
across all material. Once raw scores are converted to 
standard scores, one need not be concerned in further 
interpretation with either the battery or the form from 
which the rav7 scores came (MAT Guidelines No. 1). 

We foel that these claims are too strong for the standardization program 
that; ves used. Our principal concern is that norming and scaling were 
done on a cross-sectional, not a longitudinal, basis. In addition, the 
members of the MAT battery were equivalenced by the equipercentile method 
without regard for a student's grade. We feel that equivalencing within 
grades would have been preferable, especially in the lower grades where 
growth is rapid. 

The standardizes uion procedures that were followed have the pcte^^tial 
for failing in at least tuc ways. First, because the equivalencing of 
the various members of the battery was not made grade-specific, the ar- 
ticulation of the tests between grades may not be good. Second, the 
cross-sectional norms may mask the MAT's sensitivity to "cohort effects." 
If the test xS sensitive to such effects, the longitudinal implications 
of the norms may not apply to either the norm sample children or the PIP 
children. For example, children in the eighth grade MAT sample may well 
have had educational experiences in the elementary grades much different 



Only fall data were used in the standardization program. 
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from the experiences of the children in the sample's elementary schools. 
This would mean that the growth curve implicit in the MAT norms is not 
what would have been observed had the norm group's second graders,^ for 
instance, been followed. 

There is evidence consistent with the existence of both poor artic- 
ulation between members of the MAT battery and the existence of cohort 
effects in the MAT norms. Pelavin and Barker (1976) examined the artic- 
ulation between the various members of the MAT test battery by testing 
disadvantaged children twice within seven days on members of the MAT 
series. Since no large change in the children's achievement could be 
expected in a seven-day period, one would anticipate that a given child 
would get the same standard score both times, even if he took a different 
test each time. Pelavin and Barker found only weak evidence that disad- 
vantaged children get the same score both times. On this evidence, they 
conclude that evaluators who are predominantly concerned with education- 
ally disadvantaged students should base their evaluation on something 
other than standardized test scores. 

Further support for their conclusion can be obtained from the MAT 
raw-score to standard-score convernsion tables. Figure 2-1 shows selected 
equipercentile standard-score growth curves for the Total Reading sub- 
scales of the MAT. The test batteries and subscales are described in 
more detail in Section 3. In Figure 2-1 the growth curves change abruptly 
between the firs.t and second and the seventh and <f:ighth grades for Total 
Reading. Presumably, the "sampling error" in these graphs is small, so 
we should regard the changes as real.* Is this how achievement scores 
change, or are the fluctuations cohort effects? We note that whatever 
the fluctuations are, they are not uniform across percentiles. ror 
example, in the Total Reading curves, there are some interesting dips 
between the fourth and sixth grades for the curves greater than the 80th 
percentile; these dips do not appear on the lower equipercentile curves. 
A curious feature of Total Reading score growth in the seventh grade is 
that for children of the 50th percentile or less no growth is expected 
until the summer, at which time the curves fairly shoot up on the standard 
score scale. 

On the whole, "summer growth" for Total Reading is about as large 
as the "school year growth." This means that the growth during May and 
early June, plus the growth in September and early October, is nearly 
equal to the growth in the seven months of instruction bo-tween October 
and the following May. If there is not an abrupt change in the learning 
rate in May or October, we must conclude that there is significant growth 
in most MAT-reUvant skills when there is no school. Expressed in another 



Whether the chnnj^e.s are real or not, the norm-referenced cinalyris treats 
them as real. L'he variations are assumed to be valid and exact iy known* 
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way^ the MAT is measuring skill3 that grow about as fast out of school 
as in. 

Table 2-1 shows the average of the ratios of spring to fall growth 
to the following fall to .sprint^ growth for Total Reading and Total Math 
for selected percentiles. The table shows great variation in the average 



Table 2-1 

AVERAGE OF THE RATIOS OF SPRING-TO-FALL GROWTH 
TO TOE FOLLOWING FALL-TO- SPRING GROWTH 
FOR TOTAL READING AND TOTAL MATH, 
FOR SELECTED PERCENTILES 



Percentile 


Number of 
Ratios with 
Nonzero De- 

nomina tors 


Average Ratios 


Total 
Math 


Total 
Reading 


Math 


Reading 


1 




7 


0.106 


0.250 


4 


8 


7 


1.395 


0.454 


6 


7 


8 


1.519 


0.787 


10 


7 


8 


1.068 


0.988 


20 


7 


7 


1.047 


1.661 


50 


7 


8 


1 .242 


1.724 


80 


9 


8 


1.203 


1.025 


90 


9 


8 


0.775 


1.473 


96 


9 


8 


0.434 


0.833 


99 


1 ' 


4 


0.456 


0.872 













summer growth as a function of percentile, with the center percentiles 
being most subject to it. It is p;^.r ticalarly interesting that on the 
average about 707o more growth is fou:-d in May, June^ and September than 
in October through April. 

Since the yj\T norms are cross "«e<:li onal, we do not know whether these 
findings represent: facts or artifacts. The Coleman (1966) ^-eport and the 
Jencks (19/2) study have both shown that which school is attended does not 
influence standard reading scores very much. It seems that such findings 
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are built Into the norm tables; however, we do not know whether the findings 
are based on the way MAT skills grow, or on po<v test linkages, or on 
cohort effects. 

It is possible to discuss the problems of co'ort effects and link- 
ages in the context of the classical test theory that was used to guide 
the MAT*s cons truccion . Within the classical point of view, we could 
derive models of growth that included cohort effects, item difficulties 
and the consequent suLscale difficulties. This work would form the sta- 
tistical background for a model that repre3ented "PIP effects." However, 
we did not do tnis exercise because our analysis of the content of the 
MAT items convinced us that the MAT does not represent a good "sample" 
of the PIP curricula. 

Our content analysis, which is discussed in detail in Section 5, 
also suggests that the "trait" that the MAT measures should not be called 
"achievement" for programs that are highly individualized. Individualized 
programs are so narrowly focused that children acquire the specific skills 
that enable them to accomplish specific tasks. The concept of achievement 
underlying the MAT is oriented to general skills that may be specialized 
to increasingly difficult tasks. In the MAT system the degree to which 
a child has the skill is quantified as proportional to the difficulty of 
the items he can do. If it is true that the individualized instruction 
that is supposed to result from PIP implementation causes children to 
learn srecific skills, we would expect that the item difficulties as cal- 
culated from PIP children who had the same curricula V70uid be much dif- 
ferent ^ rom those calculated from the normative data. Unfortunately, as 
will be seen in Section 5 of this report, the number of children who have 
had similar curricala is not large enough to allow a convincing test of 
the issue. 

In concluding this discussion of the MAT norms, we note that the' 
utility oi standardized tests for evaluations is a well-discussed problem. 
The basi- issues are not empirical ones, but matters of judgment to be 
discussed in perms of the tester's objectives. Some polici»;:s require 
that educational innovations significantly affect scores relative to a 
norm group before the programs can be considered useful. In this case, 
the standardized test Is criterial no matter what the objectives of the 
particular innovations are. We consider such policies rational, if they 
are clearly and deliberately defined in terms of the test's content and 
the. composition of the test's norm group. Based on our analyses of the 
correspondence of the PIP curricula and the MAT content, discussed in 
.Section 5, we feel that the use of the MAT for evaluating the PIPs does 
not meet this condition, except in the nonempirical sense of gene' - .:est 
theory. 
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Before describing our alternative to the norm-re fereace analysis, 
we discuss technical difficulties with the "t test" (Eq. 2-2) and with 
the one-third standard deviation criterion for educational significance. 



2.3.2 Criticisms of the Norm-Referenced t Test 



The central t distribution is defined as che quotient of a stand- 
ardized normally distributed random variable and the square root of the 

9 

quotient of an independently distributed X" variable and its degrees of 
freedom. For the usual application, this would be: 




where Xj^ i == 1, . . . , n are independently and normally distributed with ex- 
pectation u and variance CJ^^ and where: 

n 



The model for the . t test described in Section 2.2 evidently would 
have Xj[ being the difference between the observed and expect^^d score for 
observation i. Formalizing the equal percentile 5core transformation, 
the following paragraphs derive a distribution for the difference * be Lween 
the average expected and the average observed spring scores. 

We rr!ay describe the processes of calculating the equipercentile score 
as follovs. Let 



zi = fsi 



_ ^ - ^ 



Z2 = ^2 (^2) = — > 

G2 



where si is the fall standard score and where and are the expected 
values and variances of the fail scores. Similarly, ^2 ^^"^^ corres- 
ponding function for the spring. 
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Assuming the standard scores are normally distributed^ the percentiles 
of s^; i=l^N are given by: 



du 



Since ^ and JLj_ are inver tiblo . v.'fc may write the formula for the equi- 
percentile standard score as: 



02 



^1 



'1 - "1 ) + "2 



For notational simplicity^ we may set 02^^! ^> then 

= b ^si - uj + U2 



The result is that the equipercentiie score is a linear function of the 
fall scores. 



In fact J the equipercentiie method of pr. J '.cling the spring score 
is the same as the usual regression method^ it is assumed that fall 
and spring scores are perfectly correlated. If this correlation^ c^ is 
positive and if b > o^ the equipercentiie expeclLition will be less than 
the regression expectation for scores less than the fall true mean. For 
in this case^ 



o < c < 1 aad cb < b 

but because 



in - ^i) < ° 

we have 

cb(s3_ -M-]_) + M.2>b(sj^ - -h \i2 

Therefore^ the equipercentiie prediction is an underes timate , from 
the regression point of view^ for scores less than the mean. If s^^ - j-ij 
> o^ the last inequality reverses. This says that^ for fall scores greater 
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than the true fall mean^ the equal percentile expectation is too large^ 
from the regression point of view. 

This bias will operate Ui make the PIPs achieve expectation more 
c?asily than if the fall-spring corrt'lation in the norms were taken into 
account . 

If we assume that the fall and spring scores are jointly normal with 
parameters u^, u^ a^, and where P is the fall-spring correlation, 

the joint distribution of the spring and equipercent ile scores is bivariate 
normal with paramet-rs Ug, \\2, bcJ^ , cj^ , and where Ug, the expected value 
of the equ iperoenti le scores, is equal to If we assume that Spj^^ ^21^' 

= 1,N are a random san:ple from the joint distribution of erpected and ob- 
. t'rved spring scores, it can be shown that 



S - s 

^N-l = ^' (2-3) 

V N 

is distributed as t on n - 1 degrees of freedom, where y '^l y P 
the usual unbiased estimates of the corresponding parameters, and where 

_ N „. N 

S2 = i E S2iand = ^ Z SeI 

1=1 1=1 

This formula differs from that given by Horst, Tal Imadge, and Wood 
by the presence of b and by the substitution of N for their N - 1. For 
moderately large samples, the effect of substituting N for N - 1 is neg- 
ligible. The effect of h may be seen by the following argument. Suppose 
b > 1. then we have the inequalities: 



u;^ 2 ^ ^ 2 



if ba^^ - ^P^i^2 ^ ^* ^^^^ 
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+ b^o^^ . 2pb 0^52 > / '^l^ ■ 2paj^a2 



Obviously, if b < 1, these inequalities reverse. Thus, whether the norm- 
referenced test is conservative or liberal depends on the ratio of the 
norm group's spring standard deviation to its fall standard deviation. 
When b > 1, the denominator of the t test is too small and will give a' 
larger Type I error rate than that reported. 

Table 2-2 shows the values of b for the MAT subtests used in the PIP 
evalua-tion. For these subtests, b tends to be about 1.05, with a minimum 
value of about O-S and a maximum value of about 1.2. These values imply 
that, if the observed variances are nearly equcl and if the. observed cor- 
relation is about 0.5, the denominator for the t statistic obtaino.d from 
the norm-referenced procedure may be anywhere from 87o too large to 117o 
too smalR On the whole, however, this error may not be important, com- 
pared with the error Introduced by uncritically taking the nonlongitudinal 
MAT standard score noms as valid estimates of fall-spring growth. 



2.3.3 Criticisms of the Criterion 

of Educationally Significant Growth 

We have mentioned that one-third standard deviation above normal 
growth is the criterion for educational significance. For the norm- 
referenced analysis, it is convenient to refer to this as "the criterion." 
However, in the PI? analysis it is not "one" criterion because, as shown 
in Table 2-2, the standard deviation of the norm group changes as a func- 
tion of time and subtest. 

Because the criterion of significance changes, the rela t j'^u.snip be- 
tween it and plausible measures of growth is not fixed, but may vary 
somewhat. For example, since average growth decreases with grade, if we 
compare the criterion for significant growth with the gain at the 50th 
percentile, we find that (l/3)a(t) is an increasing fraction of expected 
growth, unless cr(t) decreases as a function of tliTie. Table 2-2 shows 
that a tends to increase with time, but the pattern is not consistent 
because of decreases in the middle grades. However, on balance, educa- 
tionally significant growth is a IrC-rger fraction of 50th percentile growth 
at higher grades. It is not clear that V7e want our criterion for growth 
to be harder to obtain, relative to mean change, at higher grades. 

To assess the relationship between 0"( t) and the expected growth on 
Total Reading for various percentiles, a polynomial regression equation 
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Table 2-2 



RATIO OF SPRING TO FALL STANDARD DEVIATIONS FOR NATIONAL STANDARDIZATION 
GROUPS ON SELECTED MAT SUBTESTS USED IN THE PIP EVALUATION 





Word 






Math 


Math 


Math 


Total 








Knowl- 




Total 


Compu- 


Con- 


Problem 


Math 


— 


2 


Grade 


edge 


Reading; 


Reading 


tation 


cepts 


So Iving 


(Numbers) 


X 


0 


Grade 2 




















Fall SD 


11.4 


11.6 


10.9 








12.1 






Spring SD 


10.2 


11.7 


10.9 


10.0 


11.8 


12.1 


11.1 






b 


0.39 


1.01 


1.00 








0.92 


0.96 


0.06 


Grade 3 




















Fall SD 


11.4 


11.7 


11.6 


10.2 


11.8 


12.7 


11.4 






Spring SD 


12.0 


13.6 


13.0 


10.7 


12.4 


13.1 


12 .0 






b 


1.05 


1.16 


1.12 


1 .05 


1.05 


1.03 


1.05 


j.,07 


0.05 


Grade 4 




















Kail SD 


12.8 


14.5 


14.0 


10.7 


12.5 


13.1 


12.0 






Spring SD 


12.8 


14.5 


14.3 


,12.1 


12.0 


13.3 


12. 1 






b 


1.00 


1.00 


1.02 


1.13 


0.96 


1.02 


1.01 


1.02 


0.05 


Grade 5 




















Fall SD 


12.7 


14.0 


13.5 


9.9 


11.3 


12.5 


10.4 






Spring SD 


13.0 


12.4 


13.0 


11.2 


12.7 


13.0 


12.2 






b 


1.02 


0.89 


0.96 


1.13 


1.12 


1.04 


1.17 


1,05 


0-1 ^ 


Grade 6 




















Fall SD 


13.6 


15.3 


14.7 


11.6 


12.1 


13.9 


12.1 






Spring SD 


13.7 


12.6 


13,5 


12.2 


14.3 


13.6 


12.7 






b 


1.01 


0.82 


0.92 


1.05 


1.18 


0.98 


1.05 


1.00 


0.04 


Cr.idc 7 




















Fall SD 


13.5 


15.1 


14.7 


11.3 


11.9 


13.7 ■ 


12.2 






Spring SD 


14.5 


16.1 


15.8 


12.7 


12.5 


14,2 


12.8 






b 


1.07 


1.07 


1.07 


1.12 


1.05 


1.04 


1.05 


1.07 


0.01 


Grade 8 




















■ Fall SD 


15.0 


'5.9 


16.1 


13.1 


13.6 


14.7 


13.6 






Spring SD 


15.5 


17.1 


16.9 


14.4 


14.8 


15.0 


14.4 






b 


1.03 


1.03 


1.05 


1.10 


1.09 


1.02 


1.06 


1.06 


0.03 


Grade 9 




















Fall SD 


15.7 


16.4 


16.0 


14.7 


15.2 


14.7 


14.2 







Note: b -= spring divided by fall. 
SD - standard deviation. 



Source: Sp(?o-i'rt4 Report No. 8, Harcourt Brace Jovanovich, Inc., June 1971 
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was fit to the iiormative standard scores and percentiles. The basic idea 
is to simultaneously find two polynomials, Pj^(t) and P^Ct) so that for a 
standard score y: 

f(z,t) = y = P^(t) + zP^Ct) 

where z is the normal deviate corresponding to the per::encile for y. In 
this representation, P^ describes changes at the mean as a function of 
time, and Pr, describes changes in the standard deviation. 

After some preliminary runs^, a second degree polynomial was selected 
for P]^, and a fifth degree for P2. 'i'he resulting equations are: 

[lit) = P^(t) = [0.17 + (8,2 X lo'"^) t - (2.8 X 10'^) t^] X 132 

a(t) = P^Ct) = [0.12 - (7.4 X 10'^) t + (3.7 X 10'^) t^ 

- (7.3 X 10-^)t^ -f (6.3 X 10-^) t^ 

- (2 X 10-^0 ) t^] X 132 , 

where t is time in months from beginning of kindergarten. 

We have shown the coefficients to two places. The five place equa- 
tion we fit by BMD 07R has a coefficient of determination of 0.995 on 
756 error degrees of freedom- Although this coefficient is large enough 
for our present purposes^ the reader is cautl n.ed that errors as large as 
I07o can be found fairly frequently^, when predicted norm standard scores 
are compared with actual scores. Overall, however^ predicted standard 
scores for the five place equation are quite close to the actual scores 
found in the norm tables. Presumably, this good fit reflects the normal- 
izing transformation used to construct the standard scores. 

Table 2-3 shows the rates of change of the mean and standard devia- 
tion for the fitted normative standard scores for Total Reading. Compared 
with changes in the mean, changes in the standard deviation are small. 
The fitted data indicate there is a slight tendency for the standard de- 
viation to decrease at the higher grades; essentially, however, the cri- 
terion for growth--one-third the MAT norm standard devia tioii--is a constant 
criterion for all grades. Since average growth decreases as a function 



Biomedical Computer Programs (1973), pp. 387, ff, as modified by George Byrd 
of SRI. 
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Tabhi 2-3 



R^\TE OF CHANGE OF MEAN AND STANDARD DEVIATION ' ' 
FOR Fin'ED STANDARD SCORES: TOTAL READING 







Monthly 


Rate of 


Mon thly 


Rate 


of 






Mori tVy ^ 


Rate of 


Change of 


Rate of 


Change of 


Ra t io 




f iTom 


Change 




per 


Change 


a per 


OI 




K 1 n H p T — 


of Li 




Months 


of o 


Nine Months 


CO V J-y 


Grnde 


go IT tl6 n 


(1) 


(2) 


(3) 


(4) 




i 


1^ 


0 


.979 


8 


.81 


-0 


090 


-0. 


81 


-0.091 




ZD 


0 


.935 


8 


.42 


0 


065 


0. 


59 


0.071 


o 


ZD 


0 


.890 


8 


.01 


0 


133 


1 . 


20 


0 . 150 




o o 
JZ 


0 


.846 


7 


.61 


0 


.140 


1. 


26 


0. 170 


_J 


o o 

3o 


0 


.802 


7 


.22 


0 


.111 


1 . 


00 


0.14 




/. /. 


0 


.757 


6 


.81 


0 


.063 


0. 


57 


0 . 083 




30 


0 


.713 


6 


.42 


0 


.013 


0. 


12 


0.019 




56 


0 


.66S 


6 


.01 


-0 


.029 


-Q. 


26 


-0.043 


5 


62 


0 


.624 


5 


.73 


-0 


.056 


-0. 


50 


-0.087 




68 


0 


• 560 


5 


.04 


-0 


.065 


-0. 


59 


-0.117 


6 


74 


0 


.535 


4 


.82 


-0 


.057 


-0. 


51 


-0.106 




80 




.491 


4 


.42 


-0 


.C39 


-0. 


35 


-0.079 


7 


86 




.447 


4 


.02 


-0 


.019 


-0. 


17 


-0.042 




9 2. 


0 


.402 


3 


.62 


-0 


.012 


-0. 


11 


-0.030 


8 


98 


0 


.358 


3 


.22 


-0 


.035 


-0. 


32 


-0.099 




104 


0 


.314 


2 


.83 


-0 


.110 


-0. 


99 


-0.350 


9 


110 


0 


.269 


7 

! 


.42 


-0 


.262 


-2 . 


36 


-0.975 



of grade^ the criterion is an increasin^^ function of average growth. As 
already mentioned, the implication is that in term.s of average growth it 
is harder to achieve tht? criterion at higher grades- 

The following question naturally arises: If the criterion i:^ not a 
constant fraction of average growth^ is it--for a given grade--a constant 
fraction of the growth at other percentiles? To answer this question, we 
refer to our model and evaluate: 

(1/ '> )P2(^; 
dP^(t) zdP2(t) 

dt dt 
38 



for selected values of where z is the normal deviate that corresponds 
to a percentile being investigated. Table 2-3 shows dP]^(t)/dt and 
dP2(t)/dt for selected values of t in Columns (1) and (3), respectively. 

Table 2-4 shows 



dt dt 



-1 



for the second and eighth grades. We are n^electing (l/3)P7(t), since it is 
only a scale factor. Table 2-4 shows that the ratio of the criterion to the 
growth of the fitted data is relatively constant across percentiles^ within 
grades, but not betv^een grades. However, the relatively little variation 
within grades should probably not be neglected because at the second grade 
the criterion is a Larger fraction of growth of the fitted scores at low 
percentiles than at high ones, while the reverse is true at the eighth 
g^ade. la column (3) of Table 2-4, we see that the main explana tion is 



Table 2-4 

RECIPROCAL of"* THE RATE OF CHANGE 
OF FITTED STANDARD SCORES, 
FOR SELECTED PERCENTILES 
AND GRADES: TOTAL READING 
(Per Nine Months) 





Spring, 


Spring, 


Ratio 


Per- 


Second 


E igh th 


of (2) 


cen- 


Grade 


Grade 


to (1) 


tile 


(1) 


(2) . 


(3) 


10 


0 


.167 


0 


.244 


1.461 


20 


0 


.153 


0 


.273 


1.784 


30 


0 


.144 


0 


.299 


2.076 


40 


0 


.137 


0 


.325 


2.372 


50 


0 


.131 


0 


.353 


2.695 


60 


0 


.126 


0 


.387 


3.071 


70 


0 


.121 


0 


.432 


3.570 


80 


0 


.115 


0 


.500 


4.348 


90 


0 


.108 


0 


.640 


5.926 
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that, for the fitted data^ Total Reading for children at the 9uth percen- 
tile of Lne norm ^roup in the second grada is growing almost 6 times 
fnstr^r than Total Reading of the eighth grade norm children at the same 
percentile, while Total Reading of second grade children at the 10th per- 
centile is growin,*; only -ihcut 1.5 times faster. 

Thus, we find thn t the detailed answer to the question (of whether, 
for a given grade, the criterion is easier to achieve relative to some 
percentiles than to others) must be given on a grade-by-grade basis. We 
have found two grades at which the answer is affirmative. It is not clear 
that one would desire a criterion of educationally significant growth to 
be more difficult to obtain relative to some percentiles than to others, 
especially if the percentile that is more difficult depends on grade. 

However, it is important to note that the nonconstancy of the cri- 
terion just reflects changes in the norms. Since the norms are not lon- 
gitudinal, they are not strong evidence that the rate of growth for given 
percentiles changes as a function of grade, or that the ranking of rates 
for percentiles reverses for selected grades. It may be that the phenom- 
ena are just apparent and that a constant criterion is, in fact, reason- 
able. Nevertheless, as already remarked, the norm-referenced analyses 
assumes these phenomena are not apparent. 

As discussed, there are several technical problems with the norm- 
referenced anal>:?is^ but the technical problems are not the reasons we 
reject, the nomi- referenced point of view. Our opinion is that the tech- 
nical problems are just symptoms, as discussed in Sc.^tion 2.4, which de- 
scribes the analysis we prefer. 

Further discussion of the properties of norm-referenced c-n^^lysis 
can be found in Section 3.8 and in a forthcoming SRI technical publica- 
tion authored by Kaskowtiz et al. Several of the resujts presented 
above were first obtained by him. 

2 . 4 JuslLf ication for the Curriculum-Referenced Evaluation 

Broadly speaking, the purpose oi evaluating an educational innovation 
is to see whether the innovation meets selected objectives. Often the 
evaluator must choose the objectives Lha t the innovation is to meet. He 
may. choose from the objectives of federal policymakers or of the innovator, 
or may choose those implied by his own values. 

We admit that our decision to reject the norm^rof erenced model is- 
based on our concept of what is a reasonable evaluation of a field cxper- • 
iment like tiie PIP field trials. The decision is based on our values, 
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not on technical points. For the convenience of the reader^ the next 
paragraphs state our biases and their implications for the design of 
evalua tions . 

We propose as a principle that an evaluation must describe what oc- 
curred. An adequate evaluation does this in enough detail to assure rea- 
sonable men that there is a direct connection between the processes of 
interest and the outcomes of interest. Our principle demands only that 
the major terms in the evaluation be defined in enough detail so that 
their relationships can be seen. For the PIP field experiment^ this 
means that 'TIP/' "MAT raw score," and related terms must be explained 
in language that allows competent educators to know what is meant. Ob- 
viously, there is nothing profound here; we are merely making our view 
explicit. 

Relative to the definition of technical terms, we propose the pos- 
itivist's "principle of abs trac tion. " This principle is a version ol 
Occaiu's Razor: Entities should not be increased without reason. The 
principle of abstraction states that whenever one desires to define an 
entity that is said to be common to a collection of entities, i' is suf- 
ficient to refer to the collection. (This -eliminates the abstract prop- 
erty; the principle of abstraction is actually a principle that does away 
with abstraction.) 

No merely formal analysis of evaluation data will satisfy our pro- 
poc^ed evaluation principles because a merely formal analysis will leave 
the main treatment and outcome terms operationally defined (e.g., 
"achievement" would be defined as whatever the MAT measures). In this 
evaluation, the norm-referenced analysis, as its name suggests, is com- 
pletely dependent on the operational interpretation of "achievement'^ as 
whatever the MAT norms measure. This interpretation is related to the 
concept of "true score," which is operationally defined as whatever the 
items that were answered correctly have in common. "True scores" defined 
in this way is just the sort of technical term that the positivist^s 
version of Occam's Razor eliminates. We feel that the technical troubles 
associated with the norm-referenced analysis are symptoms of the confusion 
that results wh: .1 we allow ourselves to focus on entities, like true 
scores, that would be eliminated by the principle of abstraction. With- 
out assuming the existence of true scores, the Thurstone technique, which 
make the cross-sectional norms appear longitudinal, loses its appeal. 

The principle of abstraction is quite radical, unless taken with a 
grain of salt. Applied to technical and to nontechnical terms, it would 
eliminate so many entities that ordinary conversation would be impossible. 
The thing called "wall paper," for instance, would be eliminated; we would 
have to refer to its more elemental properties, like its color and extent. 
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Obviously^ we do not wish to eliminate the conveniences of ordin^iry 
linguistic conventions. The principle of abstraction should be applied 
in making the definitions o^ key technical terms^ but not applied to the 
point where we lose the ability to communicate plainly. 

When these ideas are used in defining this evaluation's principal 
dependent measure, the MAT raw .>corc, we are compelled to view the MAT 
not as an encity;, but as a collection of items. When we define "PIP," 
we are compelled to view a PIP not as an entity^ but as a set of instruc- 
tions for implementing a specific project. To satisfy our first princi- 
ple, we are obligated to display those instructions that are supposed to 
make children answer the MAT items correctly^ and we must supply reasons 
for believing that responses to the MAT items are connected to the PIP 
instructions . 

The connection we assert is that^ all other things being present, 
items will be learned from a competent teacher, given an appropriate 
curriculum and ci reasonable length of exposure to it. Thus, our evalu- 
'ation of the PIPs £S they impart achievement test scores , is aimed at 
determining the connections between the PIP-spec if ied curriculum that 
was used, instructional procedures tliat were used, and the MAT items. 

If we find that MAT items (that were covered by the PIP-specif ied 
curriculum materials that were used) are not learned, we would have no 
evidence of PIP success. 

If we find that >LAT items (that were covered by the PIP-spec if ied 
curriculum mat .\als that were used) are learned, we would have evidence 
consistent with success, but we would have no proof of success. This is 
because all PIP projects, except R-3, are "pull-out" programs in which 
the children spend only a few hours a day. Therefore, the regular school 
curriculum may be responsible for any observed ga ins . 

The curriculum-referenced analysis^ therefore, has the following 
distinct activities: 

• An in-depth examination of each PIP^s curriculum and its 
ins true tional techniques . 

• A program of data collection to verify that the curriculum 
and instructional techniques were used in the field. 

• An analysis of the curriculum that was PIP-specif ied and 
used, with a determination of which items of the MAT are 
relevant to the curriculum observed in the field. 

• Data analyses that quantitatively assess the association 
between PIP implementation variables and relevant items. 
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In spite of the earnest preachings of those who subscribe to our 
evalunticn principles, some readers will assert that true scores are a 
useful construct. For them, and to complete the black-box analysis de- 
scribed in Section 2,1^ we give in Section 3 the results of the nora- 
referenced analysis and the results of a modified norm-referenced anal- 
ysis. The modified version attempts to overcome the technical problems 
that were discussed earlier, as well as several additional technical 
problems . 

Section 4 introduces the curriculum-referenced analysis with a dis- 
cussion of the activities that were directed toward defining the PIP 
instructional programs and their curricula. It also describes the re- 
sults of the field activities that were geared to discovering wha . in- 
structional components were implemented. 

Section 5 describes our analysis of the relationship of the MAT 
items to the curriculum that was PIP- specif ied and used. It also in- 
vestigates the issue of whether the tests given to the originating sites 
were more closely related to the PIP curriculum than the MAT is. We also 
discuss the possibility of analyzing children's scores with the discussion 
limited to only those items that were shown to be covered in the PIP 
program. 

Section 6 consists of formal analyses relating PIP-relevant }'IAT 
items to implementation variables. 
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3 RESULTS OF THE NORM-REFERENCED ANALYSIS 



3 . 1 Introduction 

In Section 3, we present the results of the norm-referenced analysis 
of the MAT data. The nonn-ref erenced analysis is regarded by some as the 
principal measure of PIP effectiveness. That is, if the PIP design is 
successful, the PIP projects will show educationally significant gains 
for their students, just as the originating sites did. We have examined 
the generic true-score justification for the norm-referenced analysis 
and have found that it had no empirical content; even though several 
achievement tests were used in selecting the originating sites, no actual 
sampling of tests or students was done. Since this evaluation is based 
on a single test, the MAT, the following analyses are not operationally 
equivalent to those conducted on the originating sites. For that reason 
our analyses cannot, strictly speaking, show the results that the originat- 
ing sites did. Because of this nonequivalence, as well as because of 
our distrust of the longitudinal validity of the MAT standard-score norms, 
we do not feel t'i.f^.t the results shown below adequately test PIP effec- 
tiveness . 

3.2 Test Selection 

Although strict adherence to the concepts of generic true-score 
theory implies that v-^ne need not be overly concerned about the details 
of test selection, no one recoTiimends that the generic theory be taken 
that seriou='>ly. 

Because the norm-rclereaced procedures depend on having credibla 
norms, we took care to select a test that had empirical norms for both 
fall and spring. The MAT has ^^his feature. We also took care to vest 
in October and April, the months in Vnich the MAT normative testing was 
done. Only later did we realize that only fall data were u?ed in the 
equivalencing of test batteries." The use of only part of tlie data 



Tl:i total sample of students at each grade level taking Form G of the 
MAT durir.f^ the fall standardization was used as the scaling population 
(MAT Guidelines No. 1, 1972). 
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probably accounts for the erraticism shown in Figure 2-1 and possibly 
accounts for the slight reduction in norm standard-score variance at 
the middle grades. 

We also required that the test permit out-of-level testing, since 
many of the PIP children were thought to be at least one grade level 
below their normal grade. Through the device of standard scores, the 
MAT also had this feature. However, use of standard scores is not 
equivalent to using the "raw" noirms directly. We have already observed 
that the standard scores appear quite smooth statistically (as opposed 
to graphically). It was observed that only 9 parameters account for 
essentially all of the variance in the 16 norm tables for MAT Total 
Reading. It is difficult to imagine how we could predict "raw" norms 
that well with the same number of parameters. 

Since the test Cselection was made before the PIPs were operational, 
the curriculum: t:haL would be employed in the field could not be deter- 
mined. However, we ould see that, roughly speaking, the MAT items 
covered a good range of topics in mathematics and that the reading items 
were reasonable. 

Overall, we feel that the MAT is one of the best off-the-shelf 
achievement test batteries. It certainly satisfies the requirements of 
generic true-score theory, and, for those who are not put off by the 
consequences of assuming that the MAT norms are valid longitudinally 
and known without error, the MAT provides a very good basis for a norm- 
referenced analysis . 

So that those who are unfamiliar with the MAT battery can under- 
stand the types of items designated by the subtest labels (such as 
"Mathematics Concepts"), we describe below the MAT subtests used in the 
PIP evaluation. 

• MAT Primer - -The MAT Primer was given in the fall to all 
first grade children in the study (Catch-Up, Conquest, 
PTR) and in the spring to the first grade children in 
Canton (PTR), The Primer is composed of three subtests: 
Listening for Sounds, Reading, and Numbers. The Numbers 
subtest was not applicable to PTR or Conquest, which are 
reading programs, and was therefore not administered to 
children in those projects. 

The Listening for Sounds subtest contains 39 items. 
Twenty-one items require the child to match a sound 
spoken by the tester with a picture of an object whose 
name begins or ends with that sound. Eight items require 
the child to match a spoken word with a written word. 
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The Reading subtest consists of 33 items • In 11 items the 
child must select a letter that the tester has said aloud 
from a group of four letters. Seventeen items require the 
child to select the one word out of four that "best tells" 
about a picture. The remaining five items require the child 
to select one of three sentences that best describes a 
picture. 



The Numbers e-ubtest is composed of 34 items. Twenty items 
are read to the child. These items test the child's knowl- 
edge of shapes, sizes, 1:1 correspondence, numerical recog- 
nition, money, measurement, time, place value, and number 
series. Fourteen items require the child to do some simple 
one-digit addition and subtraction problems and write the 
answer in the test booklet. 

• MAT Primary I --The MAT Primary I was given to all second 
grade students during the fall (Catch-Up and Conquest). 
In the spring it was given to all the first graders in 
the study (Catch-Up, Conquest, and PTR), except those in 
the Canton PTR project. This test consists of four sub^ 
test.s : Word Knowledge, Word Analysis, Reading, and 
Mathematics. The Word Analysis subtest was given only 
in the spring to the first graders. The Mathematics 
subtest was given only at the Catch-Up sites. 

The Word Knowledge subtest consists of 35 items; for each, 
the child is required to select from among four words the 
one word that best describes a picture. 

The Word Analysis subtest is compoced of 40 items; for 
each, the child must match a spoken word with a written 
word . 

The Reading subtest has two parts, consisting of 13 and 
29 items, respectively. In the first part of the subtest 
the student selects one of three sentences that best 
describes a picture. In the second part he must answer 
eight riddles and then read five simple paragraphs and 
answer questions about each one. 

The Mathematics subtest is also divided into two p<arts. 
The first 35 items examine the student's understanding 
of counting, money, measurement, place value, and story 
problems. The second oart, which consists of 27 items, 
tests the student's ability to add and subtract one- 
and two-digit number.'i and to solve some simple equations, 
such as : 4 + = 7 . 
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MA T Primary II --The MAT Primary II was given to all third 
graders in the fall (Catch-Up^ Conquest^ and IRIT) and to 
all second graders in the spring (Catch-Up and Conquest). 
Primary II consists of seven subtests^ five of which were 
administered to the PI.P students: Word Knowledge^ Reading^ 
Mathematics Computation^ Mathematics Concepts^ and Mathe- 
matics Problem Solving. The mathematics subtests were given 
only to students in Catch-Up^ since Conquest and IRIT are 
reading programs. 

The Word Knowledge subtest consists of 40 items. The 
first 17 require the student to select from four words the 
one that best describes a picture. The remaining items re- 
quire the student to identify synonyms and antonyms . 

The Reading subtest is also divided into two parts. The 
first 13 items require the student to choose one of three 
sentences that best describes a picture. The remaining 
31 items are questions about six simple paragraphs that 
the student must read. 

Mathematics Computation^ a subtest of 33 items^ requires the 
student to add one- and two-digit numbers^ with two and 
three addends^ multiply one-digit numbers^ and solve simple 
equations^ such as: 28 - = 19. 

The MathemaLics Concepts subtest^ as in Primary I^ tests 
the student *s knowledge of geometry^ measurement^ concepts 
of fractions, place value, number series, inequality, and 
properties of number systems. 

The Mathematics Problem Solving subtest consists of 35 items, 
about one-half of which are dictated by the tester. All 
items are simple story problems, with the exception of two 
that instead require the child to pick the correct number 
sentence from a group of four. 

MAT E Lemcntary - -The MAT Elementary was administered to 
all fourth, fiCth^ and sixth graders during the fall 
(Catch-Up^ Conquest^ llIT^ and IRIT). In the spring, the 
Elcn.cnlary was given to all third and fourth graders 
(Catch-Up^ Conquest^ and IRIT) . The Elementary consists 
of the same seven subtests as the Primary II, five of which 
were administered to the students : Word Knowledge, Read- 
ing, r-!at hematics Computation, Mathematics Concepts, and 
Mathematics Problem Solving. The Liiree math subtests were 
administered to Catch-Up anc^ HIT in the fall and only to 
Catch-l?p in the spring. 
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The Word Knowledge subtest contains 50 items that require 
the student to identify synonyms and antonyms • 

The Reading subtest is made up of 45 items that require 
the student to read some stories and to identify the main 
idea, draw inferences, and det'^rTnino word meatiings from 
the context. 

Mathematics Co^aputation is a subtest of 40 items that re- 
quire the student to perform addition, subtraction, multi- 
plication, and division. Seven items require the manipula- 
tion of decimals and fractions. Four items are simple 
mathematical sentences like -i-9 - 9. 

The Mathematics Concepts subtest contains 40 items that 
attempt to assess the student's understanding of basic 
mathematical principles and geometry. 

The Mathematics Problem Solving subtest consists of 35 
mathematics word problems. Three items require knowledge 
of how to read a chart. 

MAT Intermediate --The MAT Intermediate was given to all 
seventh grade students in the fall (Catch-Up and HIT). In 
the spring it was given to all fifth and sixth grade students 
(Catch-Up, Conquest, and HIT). The format of the Intermediate 
parallels that of he Elementary, and the same five subtests 
were administered. Again, students in Conquest did not take 
the mathematics subtests because Conquest is a reading 
program. 

The Word Knowledge and Reading subtests have the same number 
of items and the same format as the Elementary, although 
the questions tend to be somewhat more difficult. 

The three mathematics subtests in the Intermediate have more 
items that focus on fractions, percetTts, decimals, and round- 
ing than does the Elementary. 

MAT Advanced - -The MAT Advanced was given to all eighth and 
ninth grade students in the fall (Catch-Up, HIT, and R-3) . 
In the spring it was given to all seventh, eighth, and ninth 
grade students (Catch-Up, HIT, and R-3). The same five sub- 
tests were given. The format of the Advanced parallels 
that of the Intermediate, and the same numbers of items are 
included in each of the subtests, but the difficulty factor 
has been increased. 
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3.3 Test Schedulln-/ 'uvc /Administration 



The major features of the test scheduling were determined by our 
desire to test syncronously with the MAT norms and by the schedules of 
the schools that housed the PIP projects. 

The desire for staying within the MAT norm testing dates (mid- 
October and mld-Aprll) meant that only the middle cycle of IRIT projects 
could be tested. Since there were no MAT norms above the ninth grade 
(and the ninth grade norms are extrapolated), we decided not to test the 
tenth grade PIP participants In the Olean HIT project. 

Fall testing was completed between 6 October and 24 October 1975 
by "test teams" composed of a test administrator and a test monitor. 
Every available PIP participant was tested. Spring testing was completed 
between 5 April and 7 May 1976. PIP participants who were absent during 
the fall testing period, or whose tests were subsequently Invalidated, 
were not tested in the spring. The rather long testing period In the 
spring was caused by the necessity to accommodate local testing plans 
and Easter vacation. 

We attempted to complete testing within five working days at each 
PIP project, both In fall and In spring. Table 3-1 shows the fall and 
spring testing dates for each project, and the number of test teams. 

In those projects where testing could be completed within five 
working days by one test team, the SRI site visitor served as test 
administrator and the local site assistant served as test monitor. 
Where testing could not be completed within five working days by one 
test team, local personnel. In addition to the site assistant, were 
hired and trained by the site visitor to serve as test administrators 
and test monitors. One exception was Benton Harbor, where testing 
could not be completed within the designated time by one test team and 
where local conditions did not provide for proper use of more than one 
test team. An SRI floating site visitor assisted the site visitor as- 
signed to Benton Harbor. 

The test teams were trained for one or two days by the SRI site 
visitor In accordanc3 with the Manual of Procedures for Project Informa- 
tion P.:. -.cages Testing. They were then supervised each day by the site 
vislt-cr uhroughout the duration of the testing. 

The Manual of Procedures for Project Information Packages Testing 
was developed to be the reference manual for the achievement testing 
phase of the !^IP evaluation. Ttiis manual explained the duties of the 
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Table 3-1 
TEST DATES AND NUMBER OF TEST TEAMS 





Fall 1975 


Spring 


1976 


? ro i e.c t 


Test Dates 


Number 
of Test 
Teams 


Test Dates 


Number 
of Test 
Teams 


Benton Harbor 


10/6-10/9 


1.5* 


5/3-5/7 


1.5* 


Blooming ton (Catch -Up) 


10/9-10/15 


3 


4/15-4/21 


3 


Bloomington (IRIT) 


10/15-10/17 


1 


4/21-4/22 


1 


Brookport 


10/9, 10/10, 10/14 


1 


4/6-4/9 


1 


Can ton 


10/6-10/9 


4 


4/12-4/15 


4 


Charlotte 


10/6-10/9 


4 


4/12-4/15 


3 


Cleveland 


10/14-10/17 


6 


4/7-4/8 
4/12-4/14 


5 


Dallas 


10/13-10/16 


3 


4/5-4/8 


3 


Galax 


10 /'^O-IO /'?'? 


I 


4/12-4/14 


1 


Gloversville 


LO /''O-IO /"^l 


2* 


4/26-4/29 


2" 


Lake Village 


10/13-10/16 


1 


4/19-4/22 


1 


Lexington 


10/6-10/10 


3 


4/12-4/15 


3 


Lorain • 


10/13-10/16 


4 


4/20-4/23 


3 


Oklahoma City 


10/6 


1 


4/12 


1 


01 ean 


10/20-10/24 


3 


4/5-4/8 


3 


Providence Forge 


10/16-10/17 


1 


5/3-5/4 


1 


Schenectady (IRIT) 


10/21-10/22 


1 


4/27 


1 


Schenectady (R-3) 


10'17-10/19 


3 


4/29 


9 


Wayne City 


10/6-10/8 


1 


4/6-4/8 


1 



Indiciit£>s more than one SRI site visitor. 
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site visitor, test administrator, test monitor, and site assistant. It 
also described the procedures for packing and shipping completed tests 
to SRI. The spring version of the manual is shown in Appendix A. 

As shown in Table 3-1, the number of test teams for three projects 
(Cleveland, "Charlotte, and Lorain) decreased by one in the spring. We 
had decided to test in the spring only those children with valid fall 
tests, which reduced the number of test -sessions required. In the 
Schenectady R-3 project, the number of test teams was increased from 
three to nine in the spring, and the number of days of testing was re- 
duced from three to one. This was do^e to accommodate local test schedules 
and to ensure cooperation from the host school district. 

The test battery for both fall and spring consisted of the MAT (Form F) 
and one of tv70 affective tests--either the Faces Atcitude Inventory or the 
Intellectual Achievement Responsibility Scale (lAR) . In addition to the 
above test battery, a PIP and si Le-sf.acif ic Student Attitude Questionnaire 
(SAQ) was administered in the spring. A discussion of the student atti- 
tude measures appears in Appendix A to Volume One. Since five of the 
six PIPs were designed to supplement reading and/or math curriculum, only 
the reading and math tests of the we/e given. Although the sixth PL? 
(R-3) was a replacement p^-^'.grara (that is, replaced the entire curriculd ii> , 
only the reading and math 'c. ^ ore a dministered because the criteria 
for effectiveness at the ^: i^^. lating site dealt only with reading and 
math. Table 3-2 shows : ^..r. levels tested at each project and the 
test administered. 

Table 3-2 also s! ovs vV,.- . ' \T levels assigned to each grade level in 
the +alL and spring. V/j Li> ^! . xception of granas 5, 6, and 7, all fall 
tests administered we^ .: \>- els recommended by, the test publisher. 
Originally ihe plan wi : . . se the same test ?r each grade level for 
the pre- and post-test: However, when the 'alj 1975 test scores showed 
no serious "bottoming out" in t^rms of va\: srure.^, a decision was made 
to move to tie recommender. test levels for '^1 gv:/as. This <.'-i*'':n 
was made because of our desire to stay clos.? : c cmpiric^L norms and 
to prevent ceiling effects from obscuring PIP effects. An exception 
was made for the Canton PTR project. ^ *::re is no mandatory kindergarten 
in the State of Mississippi, hence th/ 'JIR students were in their first 
vear of school. Examination of each sLtidcnt's placement in the PTR 
curriculum in early spring 1976 indicater chat none of the PTR students 
would hit the ceiling of the Primer, whi:e moving up to the Primary I 
level would have been unfair to many students, who simply would not 
comprehend the Primary I question; . 
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PIP TEST PUN 
\ 

1. Fall l'^]^ 



Prnji -ct 

Providence \ 
Waynt: CUy 



BciUiin llflt :.r 
(JUversvlil" 



HIT 



IRIT 



^„ 




H'-tropolitari AchieVJ'wnc Tosts 


Mtcctlvi' 
Tests; 




Prinarv I 




Elementary 




Advanced 


FACKii 
Cruder; 
1-2 


lAR 


r,ii!.' 1 


Cradt: 2 




Crado 4 


Cradti 5 
Read Math 


Crndi' 6 
R(.Md HJith 


Road Mali) 


Mil 8 
Rodd KJth 


Cr<idp 9 
Road Ml 


i 






)( X 
K )( 


X X 
X X 
X 

A X 


X X 
X X 
X X 
X X 
Y y 

A A 


X X 
X X 
X X 
X X 

V Y 


X X 
X X 
X X 


X X 






X 
X 
X 

X 


X 
X 
X 
X 
X 


)! 
\ 

V 


X 
X 
X 


X 
X 
X 


X 
X 
X 


X 
X 
X 








X 
X 
X 


X 
X 
X 










X X 


y. X 

X X 


X X 
X X 


X X 

X X 




X 
X 






X 
X 
X 


X 
X 














X 
X 
X 




















X 
X 


















X X 
X X 
X X 
X X 






X 



ERIC 



76 



I 



Table )''! (Ciindml.Mj) 
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The MAT reading and math tests were administered in conformity with 
dirtsctions outlined in the Teacher's Directions booklet for each level of 
the test. These directions were incorporated into SRI^s Manual of Proce- 
aures for Project Information Packages Testing, The manual also pro- 
vided specific schedules for administering the reading or math tests^ or 
a combination of both, at each level. Overall, the schedules were trun- 
cated from the schedules recommended by the publisher, since we adminis- 
tered only reading and math tests. We wished to administer the tesLs in 
as short a tim-j as possible to reduce the time lost from ^instruction and 
to prevent student fatigue. Tests were administered over a period of 
one to three days, with the reading test given the first day. Tests 
were given in no more than three sittings, with an hour between sittings. 
Students receiving the Primer were tested in groups of 15 or fewer. 
Students receiving the remaining levels of the MAT were tested in regular 
class sizes, not to exceed 30 students. 

Table 3-3 provides a comparison between the recommendations provided 
by the publisher and the schedules devised by SRI for completing the 
testing at each level. The table indicates some modifications in the 
order in which subtests were administered. In a few cases, the Reading 
subtest was ^ministered before the Word Knowledge subtest to provide 
more efficient scheduling and to allow the most difficult subtest to be 
given to the lower grades when the students were the most alert. The 
table also shows that the Word .Analysis subtest of the Primary I was 
administered in the spring to provide a comparison subtest for Listening 
to Sounds on the Primer for all first grades except Canton. 

'""he practice items of each test were administered before actual 
testing^ which added five to ten minutes to the first sitting for the 
Primer, Primary I, Primary II, and Elementary levels. Although the 
schedule provided in Table 3-3 was the same for both fall and springy 
it does not show the Studeni Attitude Questionnaire, which was ndrai uistered 
in the spring. The SAQ Wc^s administered in conjunction with the affec- 
tive test in each case and added five to seven minutes to adminiciatlon 
time . 



3.4 Quality Control Procedures 

To ensure a high degree of confidence in the field-test data^ SRI 
incorporated quality control ptocedures during all phases of fall 1975 
and spring 1976 testing. Quality control began with the shipment of 
test materials and ended with the test data on computer files. 
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For a description of quality control procedures in the field, see 
the Manual of Procedures for Project Information Packages Testing in 
Appendix A. 

Formal procedures for handling materials at SRI assured quality 
control of coding, keypunching, and test scoring. 

After the tests were returned to SRI, each test booklet within a 
test carton was examined, item by itCfm, and coded for keypunching by 
trained personnel following specific instructions. This exa^iin. tion 
maintained quality control by ensuring that responses were r ,irly indi- 
cated and unambiguous for keypunching and that the contents- '.pe test 
carton were not integrated with the contents of other test :ai o/s. 
Care was taken that oding errors could be traced back to tue Individual 
responsible. 

After the review and coding of each test item, each item response 
was made machine-readable through direct keypunching from the test book- 
lets to disk. All keypunched documents were 1007o key-verified and then 
transferred onto magnetic tapes . Each key puncher identified the job( s) 
he completed and returned the test booklets and keypunched information to 
PIP project personnel. Keypunching was completed at SRI in the fall . -»d 
subcontracted to an independent firm in the spring. 

All test booklets were scored by a program written by project staff. 
Gur programs provided for built-in audit totals for the number of records 
processed. Edit checks were performed by computer to identify meaning- 
less codes and, where appropriate, to test the data for logical incon- 
sistencies. The edit checks were also performed to verify v:hat all 
entries correspond* d to the coding specifications and keypunch instruc- 
tions. Computer tests for logical consistency dealt pri~.'?rily with 
assuring that the correct sequence of subtests was being scored. 

To ensure that field-test data were properly processed into machine- 
readable form, project staff manually examined at least a 107© sample 
of each type of test battery given, at least a 107o sample frOm each 
site, and at least a 107o sample of each test carton. This entailed 
comparing, item by item, each student's test booklets with the same 
information on the raw test data file. Any errors detected were flagged, 
and a notation was made next to each error identifying it as keypunch 
or coding error. Higher than average coding error rates indicated that 
all test booklets handled by the reviewer in question should be reexamined 
This resulted in one item in the Elementary Reading subtest being com- 
pletely rescored . 



Bert Laurence of SRI designed our procedures fui* producing scored recoi s 
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Error rates were c/ilculatod in terms of coding errors, keypunching 
errors, and overall number of errors. Since a data field on the keypunch 
cards was reserved for each response and since the data field sizes 
varied depending on the test battery, the total number of 'lata fields 
manually ciK^cked was calculated. The total number of errors detected 
was then divided by t:i:o total number o& fields checked to arrive at the 
error rate, rinally, all errors detected were corrected- on the raw test 
data files. Table 3-4 shows the error rates encountered during quality 
control procedures for fall and spring testing. 



Tab] c 3-^ 

SL^t>URY OF ERRORS FOUND DURING QUALITY CONTROL 
CHKCK OF KIlVPUNCHING AND CODING 
OF MAT SUBTKSTS, FACKS TKSTS, 
AND lAR TF.STS 



f 


Fall 
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Sets of test booklets 

S'j t s of test book] e t s samp I ed 


3,491 
359 


3,491 
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Subtests ehec ked 

S u b t e St fields c h e c k cd 


822 
26,752 


1,890 
83,686 


Kevpuneh err.>rs found 
Krror rate 


0 
0 


145 
0.0017 


Coding errors found 
1 K r ro r ra i e 


9 

0.003 


9 

0.0001 


T .) t a I e r ro r ra t e 


0.003 


0.0018 



3 . 5 Invalidation of Test ^ 

As (1,. scribed in Appendix A, the test administrator was allowed to 
invalidate subtests of PIP participants on site. Subtests were invalidated 
under one or more of the following conditions: 
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• Student refused to respond throughotit most of thfr subtest 

• Student borrowed answers consistently 

• Student marked multiple answers consistent ly 

• Student became ill during the subtest 

• Student was absent 

• Student worked the wrong subtest 

• Student was in special education 

• Student had a severe physical/mental handicap 

• Other reasons specified. 

The last condition provided for any unforeseen situation that, in the 
judgment of the test administrator, was serious enough to constitute in- 
validation. 

Table 3-5 shows test invalidations, by type and number, for Total 
Reading and Total Math for each project in the fall. Table 3-6 provides 
the same information for the spring. Table 3-5 shows that, for both 
Total Reading and Total Math, most invalidations were due to absenteeism, 
followed by reasons categorized as "other." One project. Canton, had 
a significant '-.umber of invalidations in Total Reading because students 
provided multiple answers. We attribute this to their unf ami liar i ty with 
taking tests, since it was their first year of school. 

Table 3-6 shows that, for both Total Reading and Total Math, a lead- 
ing cause of spring invalidations was, again, absenteeism. However, most 
losses were due to the withdrawal of student.- from the PIP program. 
Children were classified as "withdrawn from the program" for several 
reasons. Some were simply noted as no longer in the program and' were 
lost to us. In the IRIT projects, we listed the children not assigned 
to the middle cycle as withdrawn from the middle cycle; such children 
may have been assigned to other cycles. In Gloversville Conquest, we 
found that some children who were tested had not been assigned to the 
project full time; we also coded these children as withdrawn. 

Because of various site conditions, we were unable to test a constant 
fraction of the children originally rostered in the projects. In IRIT, 
as already mentioned, the children in the first and third instructional 
cycles were not tested. In Gloversville, we were forced to invalidate 
57 tests because of poor project implementation. Finally, poor test 
administration in Cleveland forced us to invalidate 37 tests i.n the . 
fall. 
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3 > 6 Ueneralizabi Iltiy of Test Results 



We do not feel that the techniques of sampling theory statistics 
can be applied to Lhe data. As usual in this type of evaluation, no 
sampling of children, teachers, or sites was possible. We do not k.ow 
how to define either the sample space or its associated probability 
measure. 

Consequently, relative frequency procedures, Ne>Tnan-Pearson signifi- 
cance tests, and confidence intervals do not have their usual empirical 
just iLication . In particular, the norm-referenced analysis, because it 
is just such a procedure, loses its empirical justification. A further 
consequence of the inability to sample is the haphazard distributioa of 
sample sizes. If we pretend that sampling theory applies, the usual 
inferential statistics act as if we were more interested in comparisons 
with larger sample sizes. For example, the norm-referenced analysis 
will reflect a presumed interest in R-3 and a presumed disinterest in 
IRIX. 

Even had we been able to follow the canons of sampling theory 
(sampling children to be assigned to PIPs, schools, and teachers), the 
resulting generalizabi lity would not have been a very useful feature of 
our evaluation. This is because, as a result of our work, the PIPs we 
evaluated will probably not be used again. In fact, one of the main 
purposes of this report is to justify this recommendation aad to describe 
hov7 we came 'jo make it. 

* 

3 . 7 Achi -.ving Criterion Growt h 

3.7. I Nomi-Rei:i?reaced Anal ys is Results 

The results of the norm-referenced analysis are presenter' in this 
section. Table 3-7 shows the one-third standard deviation criterion for 
growth. W.J demanded that the lower limit of the 957s confidence interval 
corresponding to tlic t test''' (Eq . 2-2) be greater than the criterion 
be /ore we said that the criterion was rter. . We also demanded that the 
upper limit of this confidence interval be less than the criterion before 



Our computer pro^-ram for the analysis uses the normality assumption to 
interpolate between percentiles to calculate expected growth if an 
exact value is not found in the norm tables. The method of converting 
standard scores to percentile ranks and determining expected spring 
scores for t.ne norm-referenced analysis is described in Appendix B. 
The computer program was written by SRI's George Black. 
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v;e conclude thai Lhe cri.t^Tioii was nul ino t . Si'iniLar convenLions were 
applied to the cone ii: s .nns ai.untl noi:;;ia] i;n)v.tli. If the lower limit of 
the 95% confidence IntervcTl was greater than zero, we said that normal 
.■••.:wth was achieved. it the upper Limit: was Less than zero,- we con- 
cluded- that normal growtli was not achieved. This procedure <:ol lapses 
the two-part l^MC criterion (see page 2) into a single test. 



Table 3-7 

ONK-THIRD STANDARD DEVIATION OF TflE MA.T 
NORiM STANDARD SCORES FOR SPRING 
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Note: Values in this table for ^-/ades i-8 are derived from tables in 
the Metropolitan AchievcncfU Tei.i: Special Report No, 8, 
Summary Statistics for Nal-:.onal S tandardization Grou ps' (Harcourt 
Brace Jovanovich, Inc., New York, K-w York, June 1971). Values 
for grade 9 are compu.ted from the Stnndarrl Score to Percentile 
Rank table in the Metropolitan Achievement Tests, Teacher's 
Handbook (Advanced) (Harcourt Brace Jovanovich, Inc., New 
York, Mev; York, 1.971) , 



Pri;^:er test batterv given in Canton grade I onlv. 

4- 

Li Stoning for Sounds subtest. 
Wo rd Ana lysis subtest. 

s 

■■J 

Mathematics subtest. 
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The use of these confidence intervals does not imply that we endorse 
their relative frequency interpretation in the PIP study. As discussed, 
the conditions under which the PIP field trials were conducted make it 
difficult to justify applying the techniques of sampling theory to the 
data. 

Table 3-8 shows the results of the norm-referenced analysis for 
PIPs, by grade and subtest. This table is most appropriate for those 
wishing a global view of PIP success, since projects are not distinguished 
here. Overall, the table shows that the PIPs did not result in projects 
that produced educationally significant growth. Of the 21 PIP and grade 
combinations that provided enough data to determine improvement in Total 
Reading, all showed that criterion growth was not achieved. In two in- 
stances, there was not enough information to reach a decision. Of the 
six PIP and grade combinations that provided enough data to decide in 
Total Math, four showed that criterion growth was not achieved. In nine 
cases, there was not enough information to reach a decision. 

For grades higher than the first, the PIPs did not retard growth 
from the equipercenti le expectation. Of the 20 Total Reading and Total 
Math PIP and grade combinations shown in Table 3-8 for which there were 
enough data to decide, we concluded that normal growth was maintained in 
19, At Catch-Up grade 4, we concluded that equipercenti le growth was not 
maintained in Total Reading. 

Because the total scores are sums of the subtest scores, we were not 
surprised to see the same picture prevailing for the subtests. Generally, 
the PIPs produced projects that were more successful in producing gains 
on the mathematics subtests than on the reading subtests. 

Table 3-8 also illustrates some points already discussed. The first 
grade was expected to make very large gains compared with higher grades. 
However, the observed gains over fall were not proportionately great. 
The erratic nature of the equipercent i le growth curves (Figure 2-1) is 
apparent in that the equipercenti le expected growth was negative for some 
PIP, grade, and subtest combinations. We see this whenever the gain 
over expected is greater than the gain over fall, as in HIT eighth grade 
Math Computations. When the gain over fall equals the gain over expected, 
the equipercenti le growth is zero, as in R-3 eighth grade Total Reading. 

Except for PTR (two sites) and Catch-Up (four to five sites), the 
PIPs produced projects that showed growth consistent with the equlper- 
centile expectation. Except at the first and seventh "grades, the PIP 
projects beat the equipercenti le expectation at least as often as they 
failed it. However, at the first grade they failed it by wide mar).',ins 
in the three PIPs with a first grade. 
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As discussed in Section A, our fieldwork siiows the projects to have 
been reasonably implemented, but the six PIPs ^jvaluated failed to produce 
educationally significant growtli in tlie sense of the norm--re f erenced 
analysis. However, in the same sense, the PIP projects did generally 
achieve equipercenti le growth. 

These results do not imply that some of the projects were not success- 
ful. Tables 3-9 tiirough 3-14 show the results of analyses of variance 
applied to the standard scores of projects for a given grade--for Catch-Up, 
Conquest, HIT, IRIT, R-3, and PTR. These table? also show the basic 
stati sties that entered into the norm- re ferenced analysis shown in 
Table 3-8. Table 3-9 through 3-14 show that, with the exception of the 
Catch-Up and Conquest projects, there were project differences on Total 
Reading or Total Math in either tlie fall or spring (significant differences 
at p 0.05) at all grades for all PIPs. This suggests that the global 
norm-re fu renced analyses shown in Table 3-8 were not representative of 
all the sites for a given PIP. 

Table J^lS shows tlie results of the norm-referenced analysis conducted 
by grade, site, and subtest. In spite of the significant F tests for 
project differences, the table shows fundamentally the same picture as 
Table 3-8. Wiiere there were enough data to reach a decision, the decision 
is that criterion growth was not achieved in Total Reading. The PIP did 
b *'ter on the Total Math standard scores; here, if there was enough 
i.„ormation to reach a decision, in over one-third of the cases, the 
(U'L'ision is that criterion growth was achieved. However, the sample 

'e becomes so small for many project and grade combinations that we 

^e not able to reach a decision over half the time. 

The result drawn from Table 3-8 concerning the achievement of normal 
g!«>wth is also substantiated at the project level; in the 22 cases where 
V could reach a decision on Total Reading using the norm -re ferenced pro- 
cedure, 19 cases confirmed normal" growth . The three exceptions were all 
in the fourth grade, one of the two grades in which we gave the same 
level of the MAT in fall and spring. In the 18 caces where a decision 
could be reached on normal growth in Total Math, all 18 were favorable. 

Al::o consistent with Table 3-8 is that the PIP projects were able to 
meet the equipercentlle growth expectation, often showing large (although 
generally nonsignificant) gains over expected. For example, the IRIT 
third grade projects all showed about 3 -point gains at grade 3 for the 
Reading subtest. In Math Computation, as well ss in Reading, Brookport 
and Galax Catch-Up made similar large (but nonsignificant) gains over 
expected in both the fifth and sixth grades. 
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Ufi 


HM 


Mi 


Ho 


Unlinovn 


0.25 


•0.)5 




h 


Onknoini 


5.40 


4.05 


3.62 


Unkitm 


Yfii 


1.62 


1,01 


0.69 


Ho 


Unknovn 


6.711 


4.51 


3.27 


Unknovn 


Yei 


0,33 


-0.18 


•0,22 


Ho 


Unknovn 


5.17 


3.1? 


3.58 


Unknovn 


Yei 






(4 










64 










JO 










50 










50 










64 










48 






]M 


1.41 


1.61 


Hj 


Dnknovn 


3.15 


3.15 


2.J5 


No 


yci 


0,21 




0.45 


Unknoin 


Unknovn 


•2,26 


-2,26 


•1.30 


No 


Unknovn 


2,f3 


• 1,71 


0,84 


Unknovn 


Unknovn 


3.74 


3.07 


3,86 


Ho 


Yd 


•0,37 


■0.37 


•0,18 


No 


Unknovn 






4b 










46 










19 










19 










IS 










46 










19 






lib 


O.S? 


0.9a 


Mo 


linlmovn 




I.DO 


0.91 


No 


Unknom 


3.69 




3.19 


Unknon 


i» 


1.91 


L»I 


1.60 


No 


Unknovn 


2.B4 


1J4 


1.86 


Ho 


Unkncwi 


3.24 


2.J4 


7.81 


Ho 


Yei 


;',75 


2,75 


2,56 


Unknovn 


Yh 






63 










63 










45 










46 










44 










62 










40 






iM 


•1.31 


-1.85 


Ho 




]M 


1.51 


1.46 


k 


Unknwn 


4,33 


2.65 


2.52 


Ho 


Yni 


3.50 


J,50 


3.33 


Unknovn 


Yei 


4,41 


2,61 


2.42 


No 


Yei 


1.87 


*1.87 


2,51 


Hd 


Yei 


4.08 


4,08 


6.11 


llnknom 


Yei 






11 










n 










45 










4b 










44 










11 










40 






Vl 


m 


\M 


No 


llnkmvfl 


Ul 


■2.36 


•1.07 


k 


Unknoni 
































3.17 


1,17 


0,98 


Ho 


Unknon 
















It 










is 








































18 


















1,44 










:i,04 


5.52 




Yti 
































5,U 


2,42 


3,63 


No 


Y«i 


























6} 








































66 
















m 


-1.11 


•1.05 


h 


UnknoMi 


2.79 


•i.Ji 


•0,80 


No 


Unknon 
































3.44 


•1,23 


-1,39 


Ho 


Unknovn 
















34 










34 








































34 
















3,15 


I.J5 




k 




4.28 


4,2( 


13,10 


No 


!(i 


2,16 




14, » 


ti 


Yii 


3,35 


3.35 


12.39 


No 


Yei 


3,51 


2.58 


8.?9 


Ho 


Yei 


4.38 


3,58 


16.03 


Ho 


Yei 


3,38 


3.38 


15.96 


Ho 


Yei 






m 










)9C 










m 










710 










712 










779 










774 







Iota: See Tible 3-15 for 

'friMf'Liittflini for Sonndi; Priury ■••Uori Amlyiii. 
Priier-Nuiilieri; Friiity [••Kitheaiti.i. 
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lili 



TJble 3-5 

DESCRIPTIVE STATISTICS FOR CATCH-UP. i\ mi 



Gride 2 [n utej) 
Priuary I, fait 
PrUary II, iprinj 



{,uit 3 (4 sitP]^ 
Priaary 11, fill 
Elmncary, spring 



Grade i (5 aicej) 
U<Mnt«ry, [all 
EleneiiUry, i^m 



Grade 5 jutj) 
Elttentary. [all 
Int«rigediate, apti.<; 



Grade b (4 sit«0 
EUnenury, fall 
Interoediicc, aprin 



Grade ^ (1 site) 
Interneilute, fall 
Advanced, sptini; 



Grade & (I )ite) 
Advanced, fall 
Advanced, jprinjj 



Statistic 



Mean 

SD 
N 

F ratio 
F-S 

Mean 

SD 

H 

IMS 

m 

F ratio 
F-S 

dean 
SD 

N 

EHS 
WHS 

F ratio 
F-S 

«ean 
SD 

SilS 

m 

F ratio 
F-S 

Mean 

SD 

N 

IHS 
LUS 

F ratio 
F-S 

Mean 
SD 



SD 
N 



yofd Knowledge 



Pall 



34,55 
?,J0 
55 

I1J.6} 
49.90 
4.40* 



49.54 
^.41 
59 

W.09 
35,5/ 
3,99* 



IM 

116,60 
60,54 
1.93 



4.92 
50 

SO. 33 
20.54 
3.91* 
0.50 

52,96 

52 

!i5,I!i 
49,26 
3 36* 
0,«2 

5B.i5 
S.Ofl 
83 

122,«I 
62.41 
1,9/ 
0.75 

65.2? 
7.48 
77 

62.20 
55.65 
1.12 
0.67 

Ii7.81 
9.B3 
42 

115.39 
95.11 
1.21 
0.65 

79.60 
10.45 
5 

0.91 



Teat 



1,67 



0.16 



•1.34 



-0.51 



-0.99 



0,11 



Pall 



44,02 
7,80 
47 

83,58 
59.26 
1,41 



53.41 
7.6B 
93 

110.6! 
56.66 
1.95 



56,95 
9.64 
79 

218,90 
86,03 
2,54* 



62,87 
10.66 
47 

290.94 
101.17 
2.3B' 



Reading 



43.40 



(50) 



47.03 



(40) 



58.17 



63.21 



(73) 



67.98 



82.60 



41,62 
7,75 
50 

125.11 
55,75 
2.24 
0.48 

49.23 
10.34 
52 

47.61 
110.59 
0.43 
0.16 

55.10 
9.06 
83 

93.17 
81.49 
1,14 
0.6] 

67.12 
10.48 
7) 

93.25 
110.85 
0.84 
0.59 



11.47 
42 

417,34 
109.0P 
3,83* 
0,75 

i2,00 
13,17 
5 

0,94 



•1.B5 



0,89 



■3.79 



3.52 



0,68 



H 


th cciDptitation 


Hath Concept) 


Problem Solving 


Total Reading 


Total Until 


Fall 




P L 


t 

Teit 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Jprinj 


t 

Test 


Pali 


Exp 


Spring 


t 

Test 


Pdl 


Exp 


Spring 


t 

Test 


33.07 


47.06 


48,92 


1.93 






46.18 








46,73 




34,4! 


45,07 


44.66 


-0.58 


33.07 


48,04 


49,57 


1.35 


4.05 




6,08 








8,53 








9,19 




5,38 




5.49 




4.05 




7.46 




54 


(48) 


'•9 








49 








48 




55 


(50) 


5& 




54 


(46) 


47 




49.84 




91,26 








112.48 








149,03 




85,36 




92,64 




49.84 




116.33 




14.36 




33,33 








70,18 








79,96 




25,67 




26.11 




14.36 




51.36 




3.47* 




2.74* 








1.60 








1,86 




7,33* 




3.55* 




3.47* 




2.27 








0.14 
























0,55 








0.13 




48.38 


56,28 


57.35 


1.07 


50.63 


58.37 


55.00 


-2.56 


49,08 


54.02 


53.87 


-0.11 


45,96 


48,81 


49,79 


0.25 


51,33 


60,34 


58.23 


-1.63 


7.79 




9.13 




9.05 




9,19 




8.63 




11.77 




5,92 




8.46 




7.5C 




9.52 




58 


(46) 


49 




57 


(45) 


49 




62 


(52) 


52 




47 


(40) 


5? 




57 


(44) 


48 




235.96 




2/7. ?4 




315.61 




198.85 




179.19 




60.79 




75,20 




12(i.49 




208.03 




i77.]2 




50.92 




74,29 




69.18 




76,74 




68.98 




UJ.41 




32.24 




o8,5? 




47.71 




84.66 




4,63* 




3.74* 




4.56* 




2.59 




2.60 




0.42 




2,33 




1.76 




4,36* 




2.09 








0.58 








0.62 








153 








0.4i 








0.65 


1 


56,84 


63.41 


65.00 


1,69 


54.28 


60.24 


60.21 


-0.03 


54. /B 


61.12 


60. BO 


-0.24 


53,03 


57,95 


55.92 


-3.46 


)7,95 


65,0! 


63.54 


0.65 


S,ll 




10.53 




8.65 




8.42 




9.44 




9.89 




7.39 




8.19 




8,09 




8,74 




91 


(81) 


81 




90 


(61) 


81 


89 


(77) 


SO 




93 


(83) 


8) 




87 


(75) 


'8 




241,78 




572.84 




45.63 




32.12 1 


121,25 




135,44 




11^.0! 








181,10 




202.67 




57,50 




86.53 




76.1! 




72.94 ■ 


87.63 




'■15,91 




51,62 




t5.ii • 


59,80 




6^52 




4,20* 




6,62* 




O.60 




0.4:i 




1,38 




1,41 




2.31 




1.(3 i 


J, 03* 




2.92* 








0.62 








0,54 








0,4(1 








VI 








0.70 




62,47 




It. u 




58.65 


61,10 


67,79 


6.04 


60,90 


64.44 


69, Oj 


3.1)9 


56.30 


63.^'; 


6'. J J 




^i],90 


69,75 


75.15 


5,711 


10,15 




i,n 




10.58 




8.44 




9,87 




8,c6 


! 8,0) 








9,54 




B.'^8 




81 


(71) 


73 




61 


(70) 


12 




81 


(70; 


■1 




79 


(73) 


V 




79 


(65) 


68 




394,69 




396.66 




615.00 




180.07 




333,26 




o2,02 




13:'K^i 




79.82 




494.37 




207.91 




87,55 




58.49 




B5.37 




64.74 




85,03 




70,19 




6!, 40 




IhM 




^9,24 




5k:o 




4,50* 




6.73* 




7.20* 




2.78* 




3,92* 




2,16 




2.19 




t.fll 




7,14* 




3.70*; 






0.58 








0.53 








0,49 








0.75 








i;-,60 1 


69,52 


75.03 


81.36 


5,08 


65.34 


68.21 


74,05 


4,44 


67,33 


72.0} 


;5,io 


3.C6 


63,89 


67,79 


67.71 


-0.03 


71,09 


74,80 


c2 33 


6,09 


10,59 




9.26 




9.99 




8,57 




10,67 




9,63 




9.0) 




10.65 




9.58 




7,;:! 




44 


m 


42 




44 


(39) 


40 




45 


(38) 






46 


(39) 


4: 




44 


(3^) 


39 




313,BB 




348.15 




190.03 




254.92 




558,98 




jI.19 




217.17 




248,57 




296.86 




!91.i7 




96,98 




64.98 




93.10 




58.36 




81,39 




81,11 




72,67 




102.71 




76.33 




39.96 


1 


3,14* 




5J6* 




2.04 




i.37* 




6.8B* 




2,85 




2.92* 




2.42 




3.S9* 




4.79* 








0.63 








0.68 








0.79 








0.81 








0,83 




91,20 


94.10 


96.411 


0.^9 


93.40 


93.90 


69.20 


'0,95 


86,60 


88.60 


97,10 


\.n 


80,40 


80,40 


80.40 


0 94,20 


95,70 


98,30 


0.93 


6,14 




7,50 




13.72 




9.07 




14, !9 




8,11 




9.10 




12.62 




9.76 




6.87 
5 




5 


(5) 


5 




5 


(5) 


5 




5 


(5) 


5 




5 


(5) 


5 




5 


(5) 








0,65 








0.59 








0,19 








l.flfl 








0.65 





Exp ia the expected ndan; SD • standard deviation; N • n.»ber of atudenta [mkr in parentheaes ire the (iy™b«ri of children for «hoB both fall and sprirfi iata are svailahle); m • between ,m mi:t, WIS • within m mare: 
F-S la the fall to jprinii correlation. ' 

•SO.05. 
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Tible 3-10 
DESCRIPTIVE mnSTICS FOR H, BYCIADE 



S-3 


Statistic 


1. 


ord Kno 


wledje 






Read in 


t 1 Had) Conputition 


Hath Cuncfpti 




rroblem 


Solving 




Total R«idln!! 


total f,;tfr 


Fall 


Exp 


Spring 


T«»t 


Fell 


Exp 


Spring 


t 

Teit 


Fall 


Exp 


Sprinji 


t 

Teit 


Fait 


Sxp 


Spring 


t 

TeiC 


Fall 


Exp 


Sprii)? 


t 

Test 


Fall 


Exp 


Spring 


t ' 


Fall 


Exp 


Sprirg 


I 

lejt 


Cmde i (4 litcj) 


Hem 


SKOl 


6]J2 


6^.89 


7.05 


B0.B4 


81.20 


8$.I4 


1^.80 


90,47 


89.;b 


93.61 


14,83 


83,22 


83.78 


86.84 


12.39 


87.63 


88,85 


?I.U 


8.35 


81.02 


82.21 


85.51 


16.03 


91.93 


92.43 


95.59 


15,96 


Advanced, fill 


5D 






U.24 




14,75 




15.76 




12,33 




13.18 




12,71 




U.29 




14.32 




14.;? 




14.66 




15.51 


13,10 


13,91 


Advanced, ipring 


N 

m 


m 


(780) 


813 




m 


(7M) 


m 




933 


(786) 


816 




930 


(780) 


m 




928 


( 782) 


817 




92S 


(779) 


812 




92i. 


(774) 


812 










2624.17 




1336.53 




2427.31 




591.45 




447,94 




797,20 




mi 




191,63 




813.06 




2463,69 




2^14,24 




363.69 


346.31 








180,72 




193.7I 




213.94 




240.34 




150.72 




172,71 




1S9.5Q 




201,86 




210.84 




216,10 




207.76 




230.48 




170.86 




191,41 






F ratio 






13. 




6.25* 




10.10* 




3.92* 




2,S9 




5,00* 




4.23* 




0.91 




3.76* 




11.86* 




12. 64* 




2,13 




3.91* 






F-S 






0.S8 








0.84 
















0,85 








0.83 








0,92 






0.91 





Key: ixp i% the Mp^c^eC jcan; 
•irisj :orrelnion. 

*p'Sfl,05. 



• itmdard deviation; H " nuaher of students (miDberi in parentheiea irt the nuafceri ol children for lAion tioth fall and apring h'A m available); 8HS « between mean miiarc; WHS ■ uthin nean square; F-S is tlie fall to 



FRir 



l\iblo Ml 



DESCRIPTIVE STATISTICS FOH ITiNyiHiSl , BY flKADE 



Word Knowledce 







F.1II 


lixp 


iipriiiiJ 


t 


Fall 


Exp 


Spring 


I 

Test 


Fall 


Exp 


Sprine 


t 

Test 






U 70 
j3, r) 




^7, 1 V 


r\ 7 1 


J.. Cm 

jD.59 


44,7; 


4 j,5h 


1 , 1^} 


36,99 


46.28 


46,89 


2,09 


U^lm'kpii I fill 

rriuiary i, niii 




i i,f\ 
O.OU 








7.10 




7.56 




6. HI 




5.44 




^ r 1 tn '1 I'll t I l! I\r 1 n /I 


4* 






1 /U 




193 


(163) 


173 




1 tM 

196 


(159; 


I/O 








OJl ,1)7 




JJ.O/ 




ICT 7T 

JoJ. /J 




'> 17 ') 1 
iJj. jJ 




524. U4 




64.5/ 




















1 1 OK 




41,40 




29.21 






F ratio 






!. 1h 




B.17* 




"4.24* 




12.66* 




2.21 






F-S 














0.51 








0.69 






Mean 


/ft y n 


51.46 


52. ?0 


1.60 


44. 3B 


46.35 


50.07 


5.08 


45.77 


47.86 


49.89 


2.78 


Primary II, tnll 


SD 






7.15 




7.4H 




ti.57 




5.85 




7,77 




Eli'moDtjrv, sprinj; 


N 


11:4 


(115) 


iJfl 




177 


(138) 


138 




154 


(115) 


13b 






DUC 


J4.1/ 




55,43 




107.36 




73,70 




55.56 




89.87 






Who 






C 1 AC 

;l.l'0 




55.41 




73.36 




33.88 




59,89 






F ratio 


ll CO 








1.94 




1.00 




1.6A 




1.50 






F-S 






0.59 








0,42 








0,54 




urace ij iiicsj 


Mean 




56.71 


56. &7 


O.IJ 


50. 2S 


■^5.15 


5^.05 


-0.12 


50.02 


53.99 


54.79 


1.98 


K 1 Arhi.1 n K ^ 1^ fill 

tiemtncary, laii 


cn 






1 to 

/.5c 




7.89 




10.55 




7.28 




8.04 


r 1 tiff i^n t tfv ifnr^no 

L icii5cin.iu y , suriiiu 


fj 
ii 




UUoJ 


1 1 1 
1 1 1 




152 


(111) 


111 




149 


(108) 


III 






nrij 










203.82 




31 ♦.4! 




110.25 




89.49 












ca 1 / 




59.35 




107.45 




52.22 




64.25 






F ratio 










4.7B* 




2.9.1 




2.11 




1.39 






F-S 






0.71 








U.64 








0.73 




urcicit' ) [j siiesj 






LI \1 


65.5: 


2.42 


56,05 


62. J6 


66.91 


4.50 


56.71 


61.81 


65.41 


5,45 


Elementary, fdl 


Ch 
ou 


7 b 1 




6./9 




9.72 




8,71 




7.77 




7.5? 




Internod idtc , sprini; 


M 






ifi 
69 




96 


(69) 


69 




94 




69 






dHS 






40J.96 




835.71 




257.17 




663. 06 




417.57 






tJMZ 






jj.iU 




78. "^9 




70,35 




47,21) 




46.42 








I.Jo* 




II f 0 J; 

11. 4b* 




10.63* 




3.66* 




14.05* 




9.00* 






F-S 






0,76 








0.53 








0.72 




Grado 6 (J sites) 


Hcan 




68. ?0 


71.07 


],6? 


63.83 


70.77 


72.33 


1,74 


63.79 


bfl.45 


71.40 


4.84 


Elementary, hll 


SD 


'^.82 




8.01 




11.97 




10.81 




10.74 




9.26 




IntcmediJto, sprine 


N 


10b 


(87) 


6? 




106 


(87) 


8? 




106 


(87) 


87 






BMS 


19113.80 




1112.17 




3096,08 




1289.55 




2608.74 




I4U5.60 








59. M 




39.25 




85.89 




8^.03 




66,83 




54.35 






F v'atio 


13.21* 




2a. 33* 




36.05* 




14.49* 




:i9.oj* 




25.86* 






F-S 






0.78. 








0./2 








U.84 





Key: Exp is the ap^'Cted mn\ SD ' st.indjrd d.'viati(Mi; :i * :ui(itbi'r of stiidtMits (mimbers in p.ir.^ntho.«;i's aro tin,' niimhcrs of 

children for whon both (all and sprinp data ire availahlo); BMS » biUwei'ti ino.in square; K = wiiliin mf*,in square; F-S is the 
fall to sprinj^ correlation. 

*P '<0.05. 
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Table 3-12 



DESCRIPTIVE STATISTICS FOH IRIT, BV tHADE 









Word Knowlpdee 


Kendin? 


Total Reading 


IfilT Grade 






F-iil 


Exp 


Sorini' 


t 

Test 


Fall 






t 


Fall 




Spring 


t 

Test 


tirade 3 (3 sites) 
Fiimary II, fall 
lilenii'ntarv, spring 


SI) 
N 

m 

F rjtu) 
F^S 




W 

10.24 
8,12* 


54.02 

W) 


55,46 
67 

62.48 
31.56 
1.98 

[\M 


2.28 


46.60 
8.47 
105 

414.4') 
65.10 
6.']/* 


47.23 

(67) 


S'Lll 
7.42 
67 

249,1^ 
49.04 

0.49 


5/)2 


48.41 
6.4^ 
105 

^93.31) 
36.54 
8.03* 


50.3! 
(6b) 


52.73 
5.98 
66 

112.53 
33.27 
3.38* 
0.58 


3.63 


Grade 4 (2 nitos) 
Elt'jniflntary, f.ill 
ElmUiry, spring' 


Mean 
N 

m 
m 

F r<itio 




7.80 
58 

114.00 
^0.38 
\,]^ 


63.68 
(34) 


b'lM 
7.79 
34 

543 53 
4';.53 
1L94* 
0.70 


-1.05 


55.19 
9,0() 
57 

4.47 
83.5(1 
0.05 


60.74 
(34) 


59.53 
9,95 
34 

450,42 
HH.OI) 
5.11* 
0.61 


-0,80 


55.15 
7.67 
57 

42.20 
5<J.10 
0,71 


61.34 

(34) 


60. P 
7,93 
34 

480.10 
49.92 
9.62* 
0.8! 


-1.39 



Key: Exp is the expected mean; SI) ^ standard deviation; H " nuiuhtT nf stiidciU.s (mimln^r in paronthesos 
arc th»^ mimhcrs of children for whom both fall and spring data an.* availnbUO; \\% *- bt'twrmi irn-nii 
square; 'rt'MS * within mean liqiiare; F-S u the fall to sprini' correlation. 



Tible 3-13 

DESCRIPTIVB STATISTICS FOR HIT, Blf GRADE 



HIT Crde 


Statistic 


Word Knowledw 


Reading 


Hath Cora 


utation 


Hath Concepts 


Problera Solving 


Total Reading^ 


Total Hath 


Fail ' 




SpriiiK 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


F«ll 


Exp 


'7'in^ 


: 

Test 


Ffili 




Spring 


I 

Test 


Fall 


Exp . 


Siring 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


Gride 6: Tutors (1 site) 


Kean 




I),)) 


75.27 


1.69 


72.58 


77.27 


I),)) 


-0t99 






88.45 


no 

l./U 


i til 




75.36 


1.29 


17 to 




87 77 


n on 


tD Gt 


11 0^ 

/J.}) 


Ii,}} 


1 lit 


l/M 


04. OU 


D0iv4 


1.45 


FlpfflPnf Jirv full 


SD 


6.09 




8.00 




7.47 




9.02 




13.79 




11,93 




;3.89 




11,27 




15.b9 




11.63 




6.93 




7.88 




14.60 




10.98 




Utenediate, spring 


N 


2^ 


(221 


22 




24 


(22) 


22 




15 


(11) 


11 




15 


(11) 


11 




13 


(10) 


11 




24 


(22) 


22 




13 


(10) 


11 






H 






0.80 








0.73 
















0.94 








0.82 








0.86 








0,98 




Grade 6: Tuteej 0 (ite) 


Hean 


58.23 


62.1$ 


62.20 


0.05 


5E.47 


65,03 


65.22 


0.14 


59.54 


62,08 


72.70 


7.52 


55.55 


54.24 


64.96 


7.68 


59.00 


62.04 


66.48 


4,14 


57.09 


61.24 


62.54 


1,38 


61.48 


65.65 


72.22 


6.05 


Eleaenlsry, fall 


$D 






8.63 




9.35 




9.00 




10,26 




6.66 




3.61 




7.1S 




9.44 




8.66 




8.94 




8.81 




9.41 




7.52 




Intermediate, Aprinj; 


N 


43 


(41) 


41 




43 


(41) 


41 




26 


(25) 


27 




29 


(26) 


27 




27 


(25) 


27 




43 


(41) 


41 




25 


(23) 


27 






F-S 






0.74 








0.57 
















ft u 

0.66 
























(1 fl7 




Grade 7: Tutors (2 sites) 




bUl 


88.28 


36.54 


(1.42 


86.06 


86.71 


87.69 


1,25 


98,63 


101.69 


103.64 


2.41 


93.61 


96.11 


95,27 


■0,91 


91,36 


93.66 


100,21 


8.86 


66.89 


87.94 


88,62 


1.55 


99.12 


101.71 


105.39 


6.15 


iDtemediate, fill 




11,90 




11.99 




10.67 




13.96 




11.99 




10.43 




14.27 




12.55 




11.03 




11.08 




11,4: 




13.44 




12.34 




11,11 




Advanced, sprin? 


N 


93 


(90) 


90 




93 


(90) 


90 




76 


(74) 


74 




76 


(74) 


74 




76 


(73) 


73 




93 


(90) 


90 




76 


(72) 


72 






m 






2579.40 




1078.02 




2688.68 




4292,12 




1782,77 




4806.96 




2977.90 




2890.62 




2051.37 




1889,35 




3051.16 




3930,77 




2195.84 






m 


115,i3 




115.97 




107.62 




166.58 




87.62 




85.48 




141.42 




118.45 




84.28 




95.59 




111.14 




147,95 




101 12 




93.82 






F-ratio 


2'l,'86* 




22.24* 




10,02* 




16.14* 




48.99* 




20.66* 




32,99* 




25.14* 




34 . 30* 




21.46* 




17.00* 




20.62* 




38,87* 




23.41* 






F-S 






5,87 








0.85 








0.81 








0.81 








ft fii 
U.oij 








07 








0.90 




Grade 7: Tutces (I sites) 


Hean 


12,45 


73.19 


73.84 


0.69 


72,49 


74.17 


73.42 


-0.66 


84.04 


85.71 


89.8,^ 


3.62 


76.65 


77.39 


78.47 


0.89 


77.38 


79,73 


84.15 


3,27 


72.19 


73,49 


73.31 


-(1.22 


. 83,64 


85.85 


89.06 


3.58 


Interaediate, fall 


SD 


9,51 




10J5 




9.62 




10.34 




9.90 




9,64 




1!,06 




9.36 




10.25 




11.02 




9.55 




10.34 




9,45 




9,13 




Advanced, spiinK 


fl 


6] 


(64) 


64 




67 


(64) 


64 




52 


(50) 


51 




52 


(50) 


51 




52 


(50) 


52 




67 


(64) 


64 




50 


(48) 


51 






BUS 


!99!j.68 




2539JO 




1285.77 




676.38 




1913.86 




484 76 




1134,43 




868.58 




1400.98 




2207.84 




1939.07 




1962,04 




1662,30 




1061.46 






m 


61.1] 




76.49 




74.11 




97.70 




61.60 




84.99 




102.03 




81.47 




79.19 




79.70 




62.76 




77,03 




56.44 




63.01 






F-r«tio 


12.68* 




33.20* 




■ 17.35* 




6,92* 




31.07* 




5.70* 




11,12* 




10.66* 




17.69* 




27.70* 




30.90* 




25.47* 




29.45* 




17,16* 






F-S 






0.72 








0.58 








0,6 








n 71 

O./i 








n <io 








(\ 17 








n 7Q 




Grade 3: Tutors (2 sites) 


Hean 


P2.40 


84.59 


86.00 


1.61 


79.65 


80.54 


83,70 


2.75 


90.70 


90.10 


91.26 


0.45 


79.71 


81.63 


79.37 


-1.30 


84.85 


86.08 


87.79 


0,84 


81.15 


82.34 


85.41 


3.86 


90.55 


91.26 


90.69 


-0,18 


Advanced, fall 


SD 


13.93 




14.31 




15.04 




14.66 




7.79 




11.21 




9,59 




12.30 




8.07 




11.13 




15.06 




15.31 




7.71 




11.76 




Advanced, aprinjt 


f! 


^2 


(46) 


46 




52 


(46) 


46 




20 


(191 


19 




21 


(19) 


19 




20 


(19) 


19 




52 


m 


46 




20 


(19) 


19 






m 


5733.54 




5142.50 




5948.17 




5077,98 




215.25 




370,^ 




619,29 




1340.14 




470.87 




670.86 




6579,70 




5880.14 




440.21 




833.79 






m 


83.34 




92,44 




111.91 




104,36 




52.16 




111.24 




64.26 




81,43 




42.65 




91.65 




99,82 




105.98 




38.26 




97.88 






F-ratio 


6fl.fl0* 




55.63* 




53.15* 




48.66* 




4.13 




3,33 




9.64* 




16.46* 




11.04* 




7.32* 




65,91* 




55.48* 




11.50* 




8.52* 






F-S 






0,91 








0.86 








0.35 
















0.61 








U.!l<i 








fl tt 




Grade B: Tuteos (2 sites, 


Hean 


16,76 


78.94 


79.57 


0,98 


73.48 


75.46 


76.20 


0.91 


90,70 


86.73 


93.76 


3.89 


81,85 


81.41 


83.66 


1.80 


89,54 


90.05 


92.13 


1.66 


74,90 


76.08 


78.06 


2.81 


92.04 


91.05 


94.40 


2,56 


reading; 1 site, math) 


SD 






12.45 




1U.61 




11.06 




11.27 




10.52 




9,20 




12,29 




9.34 




11,05 




10.39 




12.12 




10.13 




11.64 




Advanced, fall 


N 


72 


(63) 


65 




71 


(63) 


66 




53 


(45) 


5C 




54 


(46) 


50 




57 


(44) 


45 




71 


(62) 


65 




52 


(40) 


45 




Advanced, spring 


BNS 


21W,26 




3468.74 




2167.95 




1466.27 




























3055,78 




2877.01 














IWS 


68,15 




102.43 




82.72 




101.35 




























55.14 




103.47 














F-ratip 


40,27* 




33.86* 




26.21* 




14,47* 




























47,07* 




27,80* 














F-S 






0.fi2 








0.67 








0.75 








0.82 








0.81 








0.65 








(1.83 




Grade 1: Tutors {2 sites) 


Hean 


06.40 


89.58 


88.57 


-1.85 


83,t!9 


85.44 


86.94 


1.46 


100,96 


105,06 


106.58 


2,5. 


89.59 


91,35 


94,85 


3,33 


97.50 


101,61 


103.81 


2.42 


85.32 


86,41 


88.63 


2.51 


101.48 


104.02 


107.60 


6.11 


Advanced, full 




13,20 




13.99 




15,94 




15.58 




12,33 




12,86 




13.23 




12,72 




13.40 




13.46 




14.92 




15.19 




12.52 




12.27 




Advanced, sprinK 


N 




(71) 


72 




^6 


(72) 


72 




53 


'45) 


48 




58 


(46) 


46 




54 


(44) 


41 




85 


(71) 


72 




54 


(40) 


43 






m 


1)542, 04 




5j80.28 




10854,94 




7774.80 




1615.55 




1165,15 




3607.85 




2408.74 




4320.17 




2342 . 94 




9484.06 




7576.26 




3174.00 




1914.53 






l.)1S 


97.47 




117.28 




127, 96 




135.13 




124.38 




143.62 




113,68 




110.85 




100.03 




133,21 




111,02 




125.M 




98.64 




107.55 






F-raUo 


67,12* 




4«.43* 




84,83* 




57. 54* 




12.99* 




8.11* 




31,74* 




21,73* 




43.19* 




17,59* 




65,43* 




60.21* 




J2,lfi* 




17.80* 






F-S 






0.90 








0,84 








0.82 








0.85 








0.85 








0.91 








0.94 




Grade 5: Tuteon (2 sitf-s) 


Henn 


73,00 


]3.(tG 


76.72 


1.96 


73,00 


75.97 


73.61 


■1.07 


























72,45 


13.89 


75.06 


0.98 










Advanced, fall 


SO 


8.24 




7.19 




10.24 




12.01 




























8,75 


[16) 


8.76 












Advanced, sprin;; 


II 


21) 


(l(i) 


18 




20 


(18) 


18 




























20 


18 














\m 


26.32 




14.61 




1.05 




61.34 




























20,84 




52.71 














70.20 




54,01) 




llO.fil 




149.3! 




























79.67 




78.64 
















0,3) 




0.27 




0.01 




0.41 




























0,76 




'1,67 














F-S 






0.9! 








0-66 
































0.H4 













Key: Exp is the expected mn; SD • standard deviation; K • number of students (nuitbers in parentheses m the nimibers of children for wh™ hnth (nil and spring data are avaiUblii); B'iS ■ between Mm si]iiflrp; WHS " yitliin m^n sijuarp; F-S is ihe full to spring correlation. 
*B'<0.05. 



Table 3-14 



DESCRIPTIVE STATISTICS FOR GRADE 1, BY PROJECT 







Listening 


for Sounds 


Word 


(novledge 


Reading 


Void Analysis 


Total Reading 


Total Hath 


Grade i Project 


Statistic 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Spring 


t 

Test 


Fall 


Exp 


Spring 


Catch-Up (4 sites) 


Mean 














in fin 








JH. j/ 
















Ji./i 




11 '^1, 






Primer, fall 


SD 














8.55 




4,32 




6.19 








6.87 












5,09 




5.45 


Primary I, spring 


N 














48 




61 


(47) 


47 








48 








kl 




'57 


(46) 


48 




BHS 


92.39 












231.10 




58.89 




113.36 








176.27 








164.42 




14,78 




137.93 




WHS 


9,0? 












62.32 




16.53 




33.10 








38.38 








38.89 




26,52 




22.26 




F ratio 


10.19* 












3.71* 




3.56* 




3.42* 








4.59* 








4,23* 




0,56 




6.19* 




F-S 






















0.31 
























0.25 


Conquest (1 site) 


Mean 


32i4 












41.90 




30.93 




42.05 


"13.50 






37.62 








41,85 










Primer, fall 


SD 


4.56 












7.56 




4.56 




4.30 








4.13 








4.32 










Primary I, spring 


N 


28 


(21) 










21 




28 


(20) 


20 








21 








20 












F-S 






















0,52 


























PTH (2 sites) 


Mean 


23.56 




29.04' 








26.95* 




23.29 




30,18 


-18.77 






29.24* 








28, 50'* 










Primer, fall 


SD 


2.28 




4.49 








7.00 




4.11 




5.63 








5.33 








6.49 










Canton-Prinier, spring 


N 


333 




168 








119 




301 


(242) 


281 








118 








114 










Dallas-Primary I, spring 


BHS 


53.47 
















367.67 




551.35 




























WHS 


5.07 
















15.69 




30.08 




























F ratio 


10.55* 
















23.44* 




18.33* 




























H 






















0,32 



























t 

Test 



-11.05 



Note: Test level was changed from Primer in fall to Priciary I in spring for all grade I projects except Canton PTR, which received Priner in both fall and spring. Total Math represents the Numbers 
subtest score for Primer and the Hath subtest for Primary I. Total Reading for spring includes Word Wedge and Reading subtests. Norms used for Primary in spring are mid-year grade 1 norms. 

Key: Exp is the expected mean; standard deviation; N ■ number of students (numbers in parentheses are the number of children for whom both fall and spring data are available); BHS " between 
mean square; WHS " within mean square; F-S is the fall to spring correlation. 

*p«<0.05. 



Canton only. 
Dallas only. 
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Bloomington 


26 


?.% 


-3.75 


-3.04 


Ko 


No 


26 


10.58 


-3.76 


-3.14 


Ko 


No 


25 


8.44 


-12.36 


-5.54 


No 


No 


Brookport 


6 


kM 


-S.86 


-8.76 


So 


Ko 


5 


8.00 


-6.50 


-3.82 


No 


No 


6 


13.33 


-8.67 


-7.21 


No 


No 


Galax 


8 


6.00 


-5.25 


-4.73 


No 


No 


8 


8.13 


-6.38 


-3.25 


No 


No 


7 


16.57 


-3.43 


-1.03 


Unknown 


Unknown 


Wayne City 


J 


9.25 


-5.55 


-2.85 


Ko 


Ko 


8 


12.87 


-4.43 


-1.47 


No 


Unknown 


8 


8.13 


-13.3a 


-15.67 


No 


Ko 


Conquest 






































Cleveland 
PIR 


21 


kM 


-12.50 


-15.06 


No 


No 


20 


10.80 


-13.70 


-13.50 


No 


No 














Canton 


165 


5.85 


-0.32 


-0.57 


No 


Unknown 


131 


7.28 


-0.18 


-0.50 


No 


Unknown 














Dallas 


112 


5.45 


-5.50 


-13.12 


No 


No 


111 


7.41 


-7.05 


-10,22 


No 


No 















BOTES: NR = number of children for whom Total Reading scores are available for both f.U and spring; NH = nuniber of children for whoni Total Math scores are available for 
both fall and spring; N = number of children for whom test scores are available for both fall and spring. 



Test level was changed from Primer in fall (Listening for Sounds subtest) to Pricsry I in spring (Word Analysis subtest) for all grade 1 projects except Canlon 
PTR, which received the Primer in both fall and spring. Gains for Total Hath are based on the Numbers subtest score for Primer and Mh subtest for Primary I. 
Noras used for Primer in spring are mid-year grade 1 norms. Norms used for Primary I in spring are end-of-year grade 1 norms. 



*Data are not shown for grades having fewer than five students with valid test data. 



It is noteworthy that these gains were made by students whose averages- 
were well below the norm group's average, fall performance. Tables 3-16 
through 3-21 show the means and corresponding interpolated percentiles 
for each project with more than four cases per grade^ by grade within 
PIP. In general we see that, as intended, the PIP projects served very 
low achievers. In Olean HIT, most children who were performing well above 
their grade's norm group average were tutors for the project. At most 
projects, most grades showed averages less than the 25th percentile of 
the norm group, with several grades showing averages as low as the 2nd" 
percentile. Exceptions were the two PTR projects, which showed percentiles 
of 38 and 40 in Canton and in Dallas, respectively. For students with 
valid tests, this is a deviation from package specifications in that PTR 
was for children in the bottom quartile. Of course, if we were to take 
into account the unmeasured performance of those children who could not 
respond to the MAT, the "true percentile" would be lower. 

A striking f e^t ciw . o^ the data in Tables 3-16 thx-Jugh 3-21 is the 
evidence for "grade cftact.o'' in the Reaa. ag subtest. Gains and losses 
in mean percentiles for Reading are sl^.own in Table 3-22 by site and 
grade. Three PIPs with seven sites had first grades, all of which showed 
fall to spring losses in the percentile of their means. In the fourth 
grade of the three PIPs with ten sites, nine showed losses; one site at 
this grade, Benton Harbor Conquest, showed a gain. In the third grade 
of the three PIPs with nine sites, eight showed gains in the percentile 
of the mean; Gloversville Conquest at this grade was stationary. In 
t\ ± fifth grade two PIPs with seven sites all showed gains. At the sixth 
grade, six of nine groups in three PIPs sho\>7ed gains in the percentile 
of their means. At the eighth grade all sites showed gains. 

Again, there may be a problem with the norming and linking of the 
members of the MAT battery. At the first and fourth grades, we see 
definite decreases across a variety of sites and instructional programs. 
At tue third, fifth, and eighth grades we see increases across a variety 
of sites and instructional programs. 

Insofar as these analyses are concerned, we can only hope that 
these effects represent some feature associat:ed with the PIPs. However, 
of the six sites with a third, fourth, and fifth grade, all but Benton 
Harbor showed gains in Reading percentile at the third and fifth grades 
and losses at the fourth. Gloversville remained stationary at the third 
grade. All five sites with a third, fourth, fifth, and sixth grade con- 
firmed the above sequence and showed sixth grade gains. 

The possibility of grade effects further confirms our case for dis- 
counting the norm-referenced analysis. However, w:thin the context of 
the norm-referenced analysis, wc must accept the effects not as 
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artifacts, but as evidence, for example, that there was something wrong 
in five out of six sites' fourth grades, which was not wrong in these 
same sites* third grades. 

At this time we know of no such grade-related problems, and v;e infer 
that the grade effects are artifacts caused by some defect in the MAT 
norms. The effects are discussed further in Section 6,3. 

We conclude on the basis of available data that by and large the PIP 
projects did not pass the norm-referenced criterion of educationally 
significant growth . 

3.7.2 Comparison with Dissemination and Review Panel Criteria 

The generally negative results presented above do not show that the 
PIP field-test projects have failed the criterion that the originating 
projects passed. As discussed in Section 2, the criterion used at the 
original sites, of necessity, changed from test to test and from grade to 
grade. That is, the "one-third standard deviation" does not represent 
a single criterion; there were multiple criteria. 

However, while with few exceptions the PIP projects have failed the 
genera] criterion set at the beginning of this evaluation, it has not 
been demonstrated that they failed the criterion of replicating the 
exemplary programs* effectiveness, since the designs for evaluating the 
exemplary programs and the PIPs were not identical. 

In the next paragraphs, we sketch what the criterion analysis for 
each of the PIPs would have looked like, had the procedures for the 
original sites been followed. 

a Catch-L?p --The original program was evaluated on 1971-72 
data a^fing the Cooperative Primary Reading Test for grades 
1, ly and 3. The project passed the one-third standard 
deviation criterion on this test at grades 1 and 2. In 
math it passed at grades 1 and 3. 

For 1971-72 data on grades 4, 5, and 6, the California 
Test of Basic Skills (CTBS) was used for evaluation 
purposes. Only grade 4 passed either reading or math. 
Grades 5 and 6 did not pass either. 

In 1972-73, the MAT was used for grades 1, 2, und 3. Ip 
an evaluation based on these data, grade 3 passed reading. 
The other grades provided inadequate data for reaching a 
decision. For this year, the CTBS was again used for 
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Table J- 16 



HWNs m mimm mmm of means for cmcH-y 

FOR FALL m SPRIliC, PROJECT AND SUtlEST 







k'ord l(nouledge 


Heading; 


total Heading 


Hath GoB] 


u'lMon 




Hiih 


Concept) 


Hath ProbUa Solving 


\otil Hmh 






Fill 


Sprinj 


Fill 


Spring 


Ffll 


Spring 


Fall 




Fall 


Sprins 


Fall 


S 


pritiR 


Fall \ 


S 


rinj 
























P:r tex- 




Ptrcfii' 




Percen* 




Ptrcei." 








Pei'cen- 




Peicen* 




Percen- 




Percent 




Percen- 




SR/NH 


Hein 


tile 


,Hean 


tile 


Hean 


tile 


He in 


tile 


Hetn 


tile 


^ean 


tile 


^eiit 


tile 


Hean 


tile 


Hein 


tile 


Hean 


tile 


Kean 


tile 


Kem 


tile 


Hean 


tile\ 


Hean 


tile 


Bloomington 




























































Cride ^ 




























































Primiry 1 




34.62 


13 






35.16 


16 






34.44 


15 






























32,85 


12 






Priairy U 








46.91 


lb 






40.19 


14 






43,41 


13 






























48 .W 


li 




22/30 


























































Priiury 11 




51.3? 


26 






45.86 


12 






H?.73 


17 






48.39 


!? 






53.4) 


34 






49.86 


18 






52.97 


20 














54.1^ 


]l 






49.23 


16 






50.36 


1? 






56.61 


t? 






56.43 


21 






54,46 


17 






56.83 ) 


lb 


Cr«4e i 


40/39 
























































\ n 






55.80 


13 


58.80 


16 


53.13 


18 


54. ?2 


12 


53.15 


18 


55.B? 


14 


^6.20 


11 


63.62 


13 


54.6? 


13 


6G.80 


14 


54.00 


10 


67,37 


13 


57,33 


9 


65.85 




Grade S 


32/26 
























































\ 


Elewnury 




61.?^ 


16 






56.22 


11 






58.22 


12 






59.50 


3 






14 10 


2 






58.23 


5 






59. b2 


1 






Interneiiiite 










16 






66.16 


1? 






65.16 


16 






70.0? 


10 






67.48 


13 






66,63 








72.35 


• 






























































Gride 1 


5/5 


























































Priniry 1 




30,20 


1 






35.60 


'1 






32.00 


10 






























30,60 


J 






Prinary 11 








46.80 


16 






40.20 


14 






43.20 


12 






























4?.20 


11 




6/4* 


























































Frioiry II 




44,1? 


i 






44.6? 


11 






M.83 


10 






































Elenentiry 








49.1? 


11 






50.^0 


!9 






48.t2 


12 


































Gride 


B/6 
























































EUnifatHy 




5], 25 


ii 


60.50 


19 


5?.I2 


26 


60.b3 


25 


55.75 


24 


59.50 


21 


62.86 


29 


70.71 


29 


57J4 


16 


59.29 


11 


59.6? 


17 


62.50 


13 


63.67 


21 


70.33 


19 


Cride ^ 


13/13 


























































Eleoeatiry 




59.31 


12 






52j; 


9 






55.75 


13 






64,15 


7 






56.15 


3 






57.29 


5 






{2,85 








Intend iiLe 








63.7? 


11 






64.00 


14 






63.00 


12 






79.77 


33 






65.46 


9 






67.6.. 


13 






'6,38 


17 


Gride 6 


8/6 
























































Elmntary 




66.50 


15 






63.13 


12 






64.38 


13 






6i,75 


3 






61.62 


4 






60,25 


3 






60, .J 


1 






[ntemediite 








68.50 


13 






23.25 


21 






70.12 


15 






81,50 


19 






70.87 


10 






71.86 


10 






80.00 


12 


Cilix 




























































Cride 1 


ID/? 


























































Primary I 




40.90 


32 






31.50 


21 






38.80 


26 






























j5.89 


10 






Prisiry II 








50.50 


29 






44. 3D 


21 






47.30 


24 






























53,75 


33 


Grade 3 


fl/6 


























































Primary II 




45,63 


10 






40.?5 


? 






42.87 


fl 






42.43 


9 






43.00 


9 






9 






46.17 


7 






Eleaeatiry 








4?. 50 


9 






4J.37 


9 






44.25 


7 






54,43 


13 






11 






49.29 


8 






54.83 


10 


Gride 4 


7/ J 
























































Elemtiry 




49.86 


9 


5.3.5? 


? 


411. ?1 


10 


51.00 


7 


46.71 


? 


^1.14 


1 


51,29 


6 


52.86 


2 


51.5? 


9 


56,2'i 


? 


50.00 


5 


53.00 




52.57 


5 


56.14 


2 


Grtdt S 


6/ 6 
























































Eleuentiry 




56.33 








54.50 


10 






54.03 


g 






57.86 


1 






56.86 


4 






59.00 


6 




60.17 


2 






laternediiie 








59.I1J 








66.50 


IS 






61.fi? 


It 






72,C0 


12 






62.29 


6 






68.83 


16 






72.33 


9 


Gride h 


8/5 


























































Elesentiry 




64.00 


10 






61. ?5 


II 






61.66 


iO 






67,11 


4 






64.00 


6 






72.56 


17 






7;.)i9 


6 






latemediate 








68.3? 


13 






71.3? 


17 






69.12 


14 






83.6? 


23 






76.11 


20 






77.56 


19 






64.00 


20 


Providjace Forj(e 




























































Crude k 


20/15 


























































ilen«ntiry 




54.25 


15 


5;. 20 


12 


52.00 


14 


54.80 


ji 


51.70 


15 


54.75 


12 


56.37 


li 


65.68 


18 


53.0D 


10 


60.58 


13 


55.8) 


12 


59.56 


10 


57.00 


9 


65.4D 


10 


Grade ^ 


18/16 


























































Eleanntary 




63.44 


19 






62.11 


19 






52.06 


20 






68.06 


16 






65.53 


15 






67.62 


19 






71,12 


14 






lateroediDte 








6?.J9 


19 






?0.39 


25 






68.29 


21 






79.00 


30 






72.D6 


26 






73.19 


24 






?9.3I 


23 


(iride 6 


18/16 


























































Eiraentiry 




54. 2S 


11 






60.3? 


10 






61. ?2 


10 






70.33 


8 






64.76 


7 






o5.31 


8 






70,75 


5 






Interned lite 








65, ?2 


8 






63.32 


6 






63.83 


1 






78.06 


11 






70.65 


10 






73.25 


13 






7?.J(1 


11 


iiayne City 




























































Grade : 


3»/3* 


























































Frimary I 




























































Prinary 11 




























































'iride ] 


4*/'.' 


























































Prioary 11 




























































Elmntary 




























































ijrade m • 


8/^ 


























































Elementary 




b0.12 


JO 


1)4.12 


2? 


58.00 


2? 


53.15 


14 


*\75 


28 


59,62 


21 


63.25 


31 


75.87 


44 


57.25 


1? 


60.63 


13 


60.25 


19 


61.3? 


II 


63.75 


21 


70,00 




Grade S 


4*/4* 


























































Elemttry 




























































In'eriufdiate 




























































Grade ^ 


5/5 




















































i 




Eieoentary 




12.40 


29 






?4.60 


32 






j:.40 


29 






ai.DO 


28 






74.60 


23 






81,20 


34 






82.40 


25 






Intermediate 










22 






;e.8o 


30 






7fi.00 


26 






92,60 


46 






84.20 


38 






84. bO 


39 




: 91.20 


4J 


Grade 1 


5/5 
























































Iflterniediite 




J8,40 


29 






82.20 


36 






80.40 


29 






91,20 


38 






93.40 


70 






86,60 


31 






94,20 


(., 






Advii^ced 








?9,60 


30 






82.00 


34 






80.40 


29 






96.40 


4? 






89,20 


50 






9J.70 


52 






98.80 


51 


Craip fl 


4*/4* 


























































Advanced 





























































NOTE: OR • Nuifber ni childreii fftr vtiom Total ReadiriR icorej are available ior both fall and iprinji; M • nunber of children (or vhoa Total Hath icorea are available (or both fill and ipring, Other lubtut seani and perrentilei uay be baied dd larger 
aaople ntet. 



ERIC 



Data are not ahown (or gradei having (ewer thin five itudenta with valid teat data. 
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Table J-17 

MhANS AND INJtSK'UTF.D I'KRCtNT I LE S OK MEANS KCR CONQUEST 
KOR FAIL ANO SPRING, BY PROJfCT AND SUBTEST 







Word Knowledge 


Read in» 


Total Reading 






K.1 1 1 


spring 


Fal 1 


Spr inkt 


Fall 


Spring 




Toc*il 


Mean 


Prrcun- 
till* 


Mean 


Percenr 
tile 


Mean 


Perct*n- 
c 1 le 


Mean 


Pcrcen- 
t i ! e 


Mean 


Percen- 
t ile 


Mean 


Purccn- 
t ile 


B e n C it n H .i r t» o r 




























Prim.irv I 
primary 11 


:3 


33.18 


1 1 






32 .M 


.! 


42 .21 


18 


32.07 


10 


45,64 


19 


G t .ul .• j 
"♦■j.+.a.-% '1 
f. I ♦■T.t-nt .ir V 


-1 




: <. 


;>1 .2b 




43. 20 


9 


4e, 12 


14 


45.08 


11 


4 7. 79 


i; 


l.r.1':.- i 

£ 1 « <n«*'.« .-r V 


:3 


■1 .07 


10 


W.58 


I 3 


50. 3tJ 


12 


1 7 . 86 


18 


49, 18 


10 


56. 54 


15 


<.r.i.l.- •) 

I t .1 r y 
I r;t f rw.' j i.i: •■ 




^H. 1 ') 


10 








10 


68. 20 


20 


57.40 


1 I 


60 . 00 


I 7 


Or.i.i.* 6 

t. l»':n.*nr a** / 
In'. t*rtn»*ii i ar »■ 


I-. 


61. '){) 


10 






03. 36 


13 


69.43 


13 


62. 21 


10 


68,07 


12 


C \ eve 1 .inil 





























l.r.v!.' : 
Pri-.irv 1 
I'r iT.tr V ! I 


los 




30 




24 


37.84 


24 


46.94 




38.!j9 


25 


47.84 


2 7 


t.ruli- j 

PriT.irv II 
L 1 f'-'fnr .1 r V 










19 


44.42 


10 


50.63 


19 


45. 75 


12 


50.23 


17 


K 1 »"Tien t «i r V 






i: 




1 1 


49. 12 


10 


52.71 


9 


49. 30 


11 


54.08 


11 


Uri.le ^ 

i. 1 .'ment .ir V 


1 : 


32. Irt 


3 


f>o.:i 


8 


49. 17 


4 


62.33 


11 


49. 71 


3 


59.82 


8 


j '<r.i.!- t 

j I:-*. r-;.<<i iJtc 




*>b ,il 


3 


6 3 , 5U 


b 


53,45 


3 


64.09 


7 


53.95 


2 


62.91 


6 


1 
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36 .4b 


21 


44 .i.2 


21 


3/. 19 


21 


45, 58 


18 


\ , . 

: i ■ ■ • . f •» t .■ 
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47.^.3 


] 5 


4a. ^3 


1 5 


47,43 
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r » I-' 
t 1 -n.'!- 1 .ir V 
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1 I 


52.63 
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S7.00 
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5tJ.26 


1 3 


55.48 


1 3 


1 

: r. . t .ir V 
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j . "r-r;.-,! I at .> 






; ^ 




23 


58. 2'i 


14 


6H. 74 


2 2 


60. 32 


17 


68. 29 


21 


1 h 1 T t n t .1 r V 


., 

) 




: 1 




26 


70.04 




76.69 


27 


69.^9 


21 


75.98 


2t] 



'J <to Ic>t<ii N • f<,Th»'r >; chiMren {or whom Total Reading !»cor<"« arc av,iilabtc for boih f«l) and sprinp. 
s!iht.«'>,; -r. ins .md p.-'r c j'n t 1 1 •« may br basrd on larger janptc nizcs. 
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grades 4, 5, and 6. Grade 5 passed reading and grades 4 
and 5 passed math. Thus^ at one time or another, all 
grades except grade 1 failed to meet the criterion. 

• Conquest " -Conquest was not examined on the basis of a 
normed test. 

• HIT -"Grade 6 was evaluated for HIT on the basis of 1971-72 
Wide Range Achievement Test data. The HIT pupils were 
judged against the mean change of the age-specific norms. 

• IRIT --IRIT was evaluated using 1972-73 data from the 
California Achievement Test. Grade j was successful. 
Data were inadequate for evaluation of fourth graders. 

• PTR--For five sites, including the originating site, 
previous FTR evaluation data are available for tutored 
and control groups » The d,^-ta show statistically signifi- 
cant results in favor of the tutored groups. 

• R-3 --R-3 was evaluated using 1970-71 and 1971-72 data on 
the CTBS . Eighth grade children's scores were examined 
and the two-year gains in reading met the criterion. 
Gains in math did not. 

Data collected in 1972-73 on seventh graders indicated 
that reading gains did not meet the criterion, but math 
gains did. 

Thus, in only one project, Catch-Up, was the MAT battery used and 
then only for grades 1 through 3 during 1972-73. Only grade 3 met the 
criterion. 

Whether the PIP projects could meet the original criteria is an 
open question. Based on the analysis (in Section 5) of the similarity 
of the original Validating tests to the MAT, it seems possible that, if 
the original programs had been tested with the MAT and had been tried 
on the PIP criteria, they might not have fared much better than the PIP 
field-test projects. 

3.7.3 Conclusion 

The claim that the six PIPs would induce projects that could pass 
the norm-referenced analysis for the achievement of educationally signifi- 
cant growth is one of the PIPs' chief features. In Section 2, we reviewed 
with the reader the foundations for this claim; we also discussed the 
consequences of uncritically accepting the one-third standard deviation 
criterion or the oquipcrccntile definition of expected growth. 
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Tflile 3-18 



mi AND INTERPOUTED PERCENTILES OF HEAKS FOR HIT 
FOR FALL AND SPRING, By PROJECT AKD SUBTEST 
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69.55 
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80.09 
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70.27 
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80.80 
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5 
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85.10 
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6 
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12 






tjj,44 


15 






Tutees 


20/8 


65.45 


] 








1 1 
11 








Q 
1 






71.75 


5 






66.12 


5 






67,30 








71 LI 


I 






Advanced 




























































Tutors 


24/9 






■'0 CI 


il 






78.62 


27 






79,17 


28 






90,44 
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90,78 
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20/8 






I5'*.)U 
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O8.60 


11 






65.10 
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16 


80.63 


15 


74.02 


12 


77.65 


12 


77.37 


12 


79.45 


13 


99.21 


31 


103.05 


40 


82.24 


13 


86.95 


20 


90.68 


18 


96.89 


24 


96,01) 


22 


101,18 


32 


Tutees 


n/0* 


73.29 


9 


76.9^. 


11 
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10 


74.06 
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89.74 
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88.36 
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96.88 
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93.83 


52 






102.03 
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44/40 


76.70 


25 






76.02 


25 






76.45 


24 






86.76 


28 






78.81 
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80,00 


18 






86.30 


19 
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66/63 






91.77 
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92.33 


61 






105.46 


83 






97.63 


75 






102.32 


67 






107.48 


77 


Tutees 


44/40 






7S.09 


26 






75. ol 


23 






77.05 


24 






91.31 


33 






80.3E 


25 






87.60 


29 






91.37 


29 


Grade 8 
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Tutors 


15/1* 


96.93 


62 


101.20 


67 


94.67 


55 


98.80 


63 


96.80 


60 


101,67 


67 


































Tutees 


45/40 


80.50 


23 


84.43 


27 


76.78 


18 


79.64 


21 


78.82 


18 


82.64 


23 


89.73 


20 


93.42 


25 


81,41 


17 


83.33 


19 


89.05 


24 


91.89 


30 


91.05 


18 


93.80 


24 
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Tutors 


31/23 


57.23 


55 


98.13 


51 


96.16 


5(1 


98.56 


50 


98.06 


54 


39.66 


, 53 


106,42 


57 


111.12 


62 


99,00 


50 


101,48 


57 


106.76 


58 


109,80 


62 


109.96 


54 


113.22 


62 


Tutees 


1/0* 



























































Note: NR " number of children for whom Total Reading scores are available for both fall and spring; NH » number of children for whom Total Hath scores are available for both fall and spring. Other subtest raeani and percentiles may be based on larger 
sample sizes. 



♦Data a-e not shown for grades havin? fever than five students with valid test data. 



Table i-19 



MKANS AND INTERPOUTED PERCENTILES 0¥ HEAtiS KOR IKIT 
FOR FALL AND SPRING, BY PROJECT AND SUBTEST 









Reading 


Total Reading 






Fa} I 


Spt 




Fal 1 


Spring 


Fal 1 


Spring 


IRIT 


Total 

s 


H^an 


P^-rcen- 
t i 1»- 


Mf an 


Porcen- 
till? 




Perccn- 
tile 




Percun- 
t i lo 


Moan 


Ptf rcen- 
ti le 


Moan 


I'ercen- 
ti U 


BloooinKCon 




























Grade 3 

Frioary tl 
Elementary 


n 




j9 


37.16 


30 


69 . 69 


2 1 


57.23 


39 


52.15 


33 


55.77 


J3 


Grade i* 

Elementary 


28 




28 


64. 32 


27 


57.46 


27 


61.21 


26 


57.46 


27 


61 . 86 


27 


Oklahona City 




























Grade 3 

Primary II 
Elementary 




50.86 


23 


56. U 


27 


46.41 


13 


52.34 


23 


48.0.. 


18 


53. 11 


24 


Schenectady 




























'■:.-jdc 3 

Priciary IL 
t iirmrnt Jry 


2!) 




13 


53. 76 






4 


49.60 


\l 


44. 72 


U 


50.72 


19 


Crade 4 

Elementary 


b 




16 


53.83 


8 


53. 33 


18 


51.67 


8 


53.00 


18 


52.00 


8 



Total N - number o! ctiiidron for whom Total Reading scores are for available for both fall and spring. Otl;er 
subtest meann and pcrcontilen may be baaed on larf^er sample sizoN. 



Table 3-20 

MEANS AND INTERPOLATED PERCENTILES OF MtANS FOR GRADE ONE 
KOR FALL AND SPRING, BY PROJECT AND SUBTEST 







Li Atening 
for Soundn 


Word* 
Analys is 




Readint; 




Numbera 


Mathemat ica 






Fal 1 


Spr 


ing 




Fal 1 


Sp 


ring 
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_ .Spring 


Project 


Total 
N 


Mean 


Percen- 
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Percon- 
t ile 
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N 


Mean 


Percen- 
t lie 
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Percen- 
t ile 


Total 
N 


Mean 


Percen- 
tile 


Mean 


Percen- 
tile 


Catch-Up 
































Blnamingt on 
Pr inier 
Primary 1 


26 


2.V 58 


42 


32. 5i 


21 


26 


2 3,2;i 


41 


33.31 


29 


.15 


23.80 


55 


32.24 


17 


Brookport 
Pr iiiicr 
Prinary I 


b 


28. 33 


73 


32. 33 


20 


5 


2 5.00 


50 


33.00 


34 


6 


25.17 


63 


38.50 


41 


Galax 
Pr imi'r 
Prinary I 




2 3.87 


35 


29.87 


14 


8 


24. 12 


47 


32.25" 


23 


7 


23.00 


52 


39.57 


44 


Wayne City 
Primer 
Priirary I 


7 


30. 57 


B6 


39.86 


63 


8 


27.50 


73 


40. 37 


54 


8 


24.2 5 


58 


32.37 


17 


Conquest 
































Cleveland 
Pritser 
Primary I 


21 


37. 76 


92 


'<7.6: 


51 


20 


31.25 


93 


4?. 05 


o2 












PTR 
































Canton 

Primer 
Pr im«r 


165 


23. 18 


29 


29.03 


28T 


131 


22. 18 


38 


29.47 


in 












Dal Us 
Primer 
Primary I 


112 


24.01 


36 


2'>.m6 


1? 


111 


24.62 


49 


32. Oi 


22 












Note: Total N " 


number 


of chi Idr^n for 


whom 9 


bt<*5t SCOT tin «re avail 


able for 


both fall ar.J spring. 


"^0 Total Reading 


score . 





♦Listening for So'inds for Canton only, 
t Mid-year (not end-of-year) percent i Ua . 
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Table 3-21 



Km ANO INTERPOUTED PERCENTILES OF MEANS FOR R-3 , 
FOR FALL ANO SPRING, BY PROJECT AND SUBTEST 



a, Reading 







Uord Knowledge 


Renii 


ing 


Total Reading 






Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


R-3 


NR 


Mean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Charlotte 
Grade 8, Advanced 


228 


33.61 


25 


87.42 


35 


63,81 


27 


88.52 


39 


84.1.? 


26 • 


88.60 


36 


Lake Village 
Grade 8, Advanced 


123 


74.43 


14 


77.72 


17 


77.15 


18 


79,07 


20 


75.22 


14 


78.20 


16 


Lorain 
Grade 8, Advanced 


271 


82.41 


27 


86.21 


31 


81.21 


23 


86,15 


32 


32.09 


23 


86.87 


32 


Schenectady 
Grade 8, Advanced 


157 


81,51 


25 


85,41 


29 


80.54 


23 


84,89 


28 


81,15 


22 


85.79 


30 



0 
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Hath Con 
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Total Hath 
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Spring 


Fall 


Spring 


Fall 


Spring_ 
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Spring 


R-3 


B 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
. tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Hean 


Percen- 
tile 


Charlotte 
Grade 8, Advanced 


214 


89, 5C 


19 


94,48 


28 


84.17 


22 


87.42 


29 


87.92 


22 


91, W 


33 


91,99 


20 


96.16 


28 


Lake Village 
Grade 3, Advanced 


120 


89,4^' 


19 


90.78 


23 


81,41 


16 


83.25 


19 


36.43 


19 


87.66 


21 


90,52 


17 


92.02 


20 


Lorain 
Grade 8, Advanced 


285 


91.90 


24 


94,57 


28 


85.60 


25 


88.63 


31 


88.76 


24 


92.79 


31 


93.81 


24 


97.22 


32 


Schenectady 
Grade 3, Advanced 


155 


92.47 


25 


94,00 


26 


81.77 


17 


87.00 


28 


87,57 


21 


91,11 


26 


<AM 
.,_ i 


20 

L.I 1 ■ , 1 


95.67 


27 



NOTE: NR ■ Number of children for «hora Total Reading scores are available for both fall and spring; (W = number of children [or whom Total Hath scores 
are availitle for both fsU and spring. Other subtest ueans and percentiles my be based on larger sample sizes. 
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Table 3-22 



SIGN OF FALL-SPRING CHANGE IN PERCENTILE OF THE MEAN 
FOR READING SUBTEST, BY SITE AND GRADE 



Pro ]ect 


Grade 


1 


2 
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5 


6 
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Catch-Up 
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Brookport 




















Ga lax 
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Co nque s t 
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Gloversvi 1 le 






0 














HIT 




















Lexington 




















Tutors 




















Tutees 




















Olean 




















Tutors 


















- 


Tutees 














- 






IRIT 




















Bloomington 








- 












Oklahoma City 




















Schenectady 




















PTR 




















Canton 




















Da lias 




















R-3 




















Charlotte 




















Lake Village 




















Lorain 




















Schenectady 





















Note: - = loss in mean percentile fall to spring 
+ = gain in mean percentile fall t:o spring 
0 = no change in mean percentile fall to spring. 
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In Section 3,8 we display a modified norm-referenced analysis that 
deals with some of the shortcomings of the unmodified analysis. 

Perhaps no analysis but the unmodified analysis will satisfy some 
critics of the PIP concept. We point these critics to the arguments in 
Section 2, which show that passing the norm-referenced analysis and the 
criterion of educationally significant growth is not a compelling reason 
for justification of packaging. 

We also point out that with a less conservative approach to the 
norm-referenced analysis, we would find some evidence for the success 
of the packaging concept. For example, all third grade IRIT projects 
showed good gains on the MAT Reading subtest. On the same subtest the 
Cleveland Conquest project showed fairly large gains over expected growth 
at the third, fifth, and sixth grades, and Benton Harbor Conquest did 
well at all grades. Furthermore, when as cautious a procedure as one- 
tailed confidence intervals confirms that a^ least normal growth is 
achieved in almost every case having enough data to decide, we may con- 
clude that packaged projects have the potential for maintaining more 
than equipercentl le growth. 

3 . 8 Special Empirical Studies of the Norm-Referenced Analysis 

Daring the first year of the evaluation of the PIP field test, 
several peculiar properties of the norm-referenced procedure became 
apparent. Questions were raised regarding the assumption of equal per- 
centile growth as the basis of the normal growth criterion and regarding 
the stringency of the criterion fjov educationally significant growth. 
The validity of the equal percentile assumption was exan^ined to a limited 
extent, and the results tended to confirm the "straggler hypothesis," 
that is, that PIP participants may represent the kinds of students who ' 
lose ground over ti.p,e. Examination of the stringency of the criterion 
of educational significance indicated that gains necessary to attain 
educational signi f icance--re lative to gains necessary to attain normal 
growth--increase as a function of grade level. 

As 1 consequence cf these findings, SRI conducued a detailed in- 
vestigation of the properties of the norm-referenced procedure as employed 
la the PIP evaluation. The examination included not only the equal per- 
centile assumption and the stringency of criteria, but also the statis- 
tical properties of the procedure. Proper .les of interest were the 
sensitivity of the procedure to the unit of analysis and the effects of 
the use of the standard deviation of the difference between fall and 
spring observed scores to approximate the standard deviation of the dif- 
ference between the spring observed and expected scores. 
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The detailed results of the study are given in a working paper en- 
titled »'A Study of the Norm-Referenced Procedure as Applied to the Eval- 
uation of Project Information Packages^' (Kaskowitz and Norwood, 1976). 
The purpose of this section is to summarize the results of the study 
and indicate some of the implications for interpreting the results of 
the norm-referenced analysis. 



3.8.1 The Equal Percentile Assumption 

The p jjor assumption of the normal growth model that was incorpo- 
rated in the norm-referenced procedure is that, ceteris paribus , the 
norm percentile score of a child, class, or site will, on the average, 
stay the same between pretest and posttest. The sample of children used 
in the standardization and normiag of the MAT was designed to be repre- 
sentative of the entire school population. However, the children in- 
cluded in compensatory education programs, such as the PIPs, are usually 
different from the entire school population with respect to demographic 
and socioeconomic characteristics: They are more likely to be members 
of a minority; they tend to be from lower income families; and they tend 
to ha^ye low pretest scores on standardized achievement tests. More sig- 
nificantly, evidence from the first-year PIP evaluation indicated that 
they may be representative of students who lose ground over time^ rela- 
tive to the norm population. If this were the case, use of the equal 
percentile assumption would lead to overestimates of the expected post- 
test scores used in the test for normal and educationally significant 
growth. One might then conclude that programs were not effective in 
raising scores on standardized tests, when in fact they were. 

The study of the equal percentile assumption entailed examination 
of several large-scale data bases contrainin^; longitudinal MAT test data 
on children who would ordinarily qualify for educational programs such 
as the PIPs. Included were Follow Through (FT) project evaluation data 
obtained from SRI and Compensatory Reading (CR) program evaluation data 
obtained from the Educational Testing Service (ETS) .* In addition, data 
from a subset of the MAT norm group were obtained from Psychological Cor- 
poration. The norm data consisted of longitudinal information on a large 
subset of children who were tested both in the fall and spring standardi- 
zation programs. 



The Follow Through and Compensatory Reading programs are funded by USOE. 
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From the FT evaluation;, data on a subset of children in the compar- 
ison group--Non-Follow Through (NFT)--were used in the analysis. These 
were children who had entered kindergarten in fall 1971 and had been 
tested in at least two of the subsequent three spring test periods (1973; 
1974; or 1975) when the MAT had been administered. From the CR evalua- 
tion of children in grades 1, 4^ and 6^ three groups were of particular 
interest: children in compensatory reading programs who were partici- 
pating in the federal school lunch program (CR/SL); children in compen- 
satory reading programs who were not participating in the federal school 
lunch program (CR/NSL)., and children in schools that had no compensatory 
reading program who were participating in the federal school lunch pro- 
gram (NCR/SL). For individual children^ participation in the federal 
school lunch program was the only available indicator of socioeconomic 
status. About 73X of the CR/SL children were in schools where compensa- 
tory reading programs were funded to some extent by Title I. and about 
58% of the CR/NSL children were in schools where the compen:. tory reading 
programs were funded to some extent by Title I. These two groups^ then^ 
consisted largely of children similar to those for whom the PIPs are tar- 
geted. 

The subpopu la tion of the MAT norm group was obtained to examine the 
relationship between the gains predicted from the cross-sectional stan- 
dardization design used to derive the percentile norms and the gains em- 
pirically obtained from a longitudinal sample. These data were initially 
analyzed by Dr. Michael Beck of Psychological Corporation^ who reported 
the results in a paper presented at the 1975 Convention of the National 
Council on Measurement in Education (Beck^ 1975) . 

Two major findings emerged from the examination of the data: 

• Expected posttest scores based on the equal percentile as- 
sumption tend to be too low for students with extremely 
low pretest scores. 

• Expected posttest scores based on the equal percentile as- 
sumption tend to be too high for disadvantaged students^ 
especially disadvantaged minority students whose pretest 
scores are not extremely low. 

The first point is illustrated in Figure 3-1, which is a plot of the 
empirical growth curve of the CR/SL group on the MAT Total Reading between 
fcai and spring of fourth grade. The empirical growth curve^ indicated 
by Che dots in the figure, consists of the posttest mean standard score 
for each value of the pretest standard score. The solid line represents 
the relationship between pretest and posttest standard scores under the 
equal percentile growth ijssumption. The empirical growth curve for the 
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PRETEST STANDARD SCORE 

FIGURE S-l EMPIRICAL GROWTH CURVE FOR THE FOURTH GRADE CR/SL GROUP ON 
MAT TOTAL READING: FALL PRETEST, SPRING POSTTEST 
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CR/SL group tends *-o track the equal percentile curve in the midrange of 
fall scores, bets,, m standard scores of 46 (the 6th percentile) and 88 
(the 94th percentile). However, the equal percentile curve shows that 
students scoring below tjie 6th percentile in the fall tend to exhibit 
gains that are greater than expected in the spring. 

The second point, regarding gains for disadvantaged children, is . 
illustrated in Figures 3-2 and 3-3. Figure 3-2 shows the empirical growth 
curve for the NFT group on Total Reading between the spring of first grade 
and the spring of second grade. Aga in, the equal percentile model under- 
predicts the gains made by pupils with extremely low pretest scores. 
However, it also overpredicts gains for children, with pretest standard 
scores above the extremely low level, as indicated by the empirical growth 
curve lying below the equal percentile curve for pretest standard scores 
above 25- Even with the phenomenon of higher than expected gains for 
children with loW pretest scores, the NFT group had an average drop in 
percentile rank of 6.9 points between the spring of first and second grade 
and 2.4 points between the spring of second and third grade on Total Read- 
ing. For Total Math the average drop was 6.3 percentile points between 
the spring of first and second grade, but no drop between second and third 
grade. 

The CR evaluation groups showed gains in their percentile ranks on 
Total Reading between fall and spring across the three grade levels in- 
cluded in the evaluation. The gains averaged about 10 points for second 
grade, between 3 and 5 points for fourth grade, and about 2 points for 
sixth grade. 

The dramatic decline la percentile ranks for the NFT group over the 
two-year period supports the straggler hypothesis--that is, in the ab- 
sence of intervention programs, children targei:ed for compensatory edu- 
cation programs for the disadvantaged will lose ground relative to the 
norm group. The data from the CR evaluation, on the other hand, are not 
uiiequivocal in denying the straggler hypothesis. For one thing, children 
in the CR/SL and CR/NSL groups v;ere participating in compensatory reading 
programs that in lact may have reversed a decline in percentiles. The 
NCR/SL group that: appeared to serve the role of comparison group had per- 
centile gains that equaled or exceeded those of the two CR groups. Fur- 
thermore, the NCR/SL group has characteristics that would call into ques- 
tion the assumption that these children are typical of those who would 
be In compensatory programs. They are disproportionately from the South, 
from moderate-size cities, and of nonminority status. Finally, none of 
the NCR/SL students, as contrasted with 447o of the CR/SL students, were 
in schools where the estimated percent of students from families receiving 
public assistance exceeded 257o of the school population. 
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20 40 60 80 100 120 

PRETEST STANDARD SCORE: GRADE 1 



FIGURE 3-2 EMPIRICAL GROWTH CURVE OF THE NFT GROUP, FIRST TO SECOND GRADE. 
ON MAT TOTAL READING: SPRING PRETEST AND POSTTEST 
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20 40 60 80 100 120 

PRETEST STANDARD SCORE 



FIGURE 3-3 EMPIRICAL GROWTH CURVE FOR THE FOURTH GRADE CR/SL GROUP ON 
MAT TOTAL READING, BY MINORITY STATUS: FALL PRETEST. SPRING 
POSTTEST 
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It would appear that out-of-level testing probably does not explain 
the extraordinary gains in second grade. Pelavin and Barker (1976) in 
fact found that when pupils are tested within a short time period on both 
the Primary I and Primary II Reading subtests that standard scores on 
Primary II tend to be higher than standard scores on Primary I. This 
would mean that the out-of-level testing in the spring may have suppressed 
the gains in s'tandard scores on Total Reading. 

Figure 3-3 is illustrative of the relative performance of white and 
minority pupils. It shows the empirical growth curves for children in 
the fourth grade CR/SL group on Total Reading; white children are indi- 
cated by the broken line and minority children by the solid line. Be- 
Lw-en the pretest standard scores of 40 (the 2nd percentile) and 65 (the 
48th percentile), the empirical growth curve for minority students is 
consistently below that for white students. For the NFT group and each 
CR group, this was a consistent pattern for Total Reading and Total Math: 
Minority students consistently gained less in percentile ranks (or lost 
more) than did white students. Of course, minority status was confounded 
with other demographic and socioeconomic characteristics, such as median 
income, size of place, and region, so that minority status of itself can- 
not be identified as the sole factor in accounting for differences in 
growth. Nevertheless, since so many of the disadvantaged children tar- 
geted for programs such as the PIPs are of minority status, the differ- 
ences in performance are of particular Interest and import. 

3.8.2 The Statistical Proper tic^S'-of the Procedure 

Two questions were examined regarding the statistical properties of 
the norm-referenced procedure: (1) How sensitive is the test to the unit 
of analysis? and (2) How good an approximation is the standard deviation 
of the difference between the pretest and posttest scores to the standard 
deviation of the difference between the posttest and expected posttest 



Algebraically, the t statistic used to test for normal growth may be 
e;-:pressed as follows: 



scores? 
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where 



mean standard score on the posttest 

mean s tandard score on the pre tes t 

sample variance of the difference between pre- and post- 
test standard scores 

number of cases 

function that gives the '^expected" posttest score^ given 
the pref.est score. 

The current procedure converts the pretest mean score to a predicted^ 
posttest mean score. The conversion could also be made individually for 
each student's score; the individual predicted scores could then be ag- 
gregated to yield the average posttest predicted score. Since the func- 
tion that converts standard scores to percentiles is intended for use at 
the individual student level, the question arises whether this alterna- 
tive might yield different results. If the transform, f, is linear, there 
would be no difference between the two procedures. In the criterion 
study, it was found that under the equal percentile assumption the func- 
tion could be closely approximated by a linear function so that cho cur- 
rent procedure is relatively insensitive to the unit of analysis. How- 
ever, under alternative nonlinear models of normal growth, different 
results can be obtained, depending on the unit of analysis selected. 

The question of the variance estimate used in the denominator was 
discussed in Section 2.3.3. Horst, Tallmadge, and Wood (1975) recommend 
using Vpost-pre estimate Vpost-'post^ where post is an estimate of the 
predicted posttest score under the normal growth assumption. The rela- 
tionship between these two variances depends on the form of the function 
that predicts the posttest score from the pretest score and the relation- 
ship between pre- and post-test scores. As indicated earlier, under the 
equal percentile assumption, the function f is^ approximately linear. 
Since the coefficient of che first-order term is close to one, the vari- 
ance of the difference between the pre- and post-tests would appear to 
be an adequate approximation. 

Four sets of data on Project Catch-Up from the first-year PIP eval- 
uation were reanalyzed to assess the impact of modifications in the sta- 
tistical procedure- on the results of the n orm-re terenced analysis. The 
transformation from fall score tu expected spring score was applied at 
the ;-:tudent level rather than at the site level- The mean and the vari- 
ance oC t'lio clLffcrcnce between the student observed and expected spring 
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post = 
X 

pre = 

V 

post -pre = 

N = 
f(x) = 



scores were used in the calculation of the t statistic. The results are 
summaried in Table 3-23. 



Table 3-23 



COMPARISON OF RESULTS OF ORIGINAL 
AND MODIFIED NORM-REFERENCED PROCEDURE 



Test and Procedure 


Number 
of 

Students 


Ga in 
over 
Expec ted 
Gain 


SD* 


t 

Test 


Meets 

il 1. lUa X. 

Growth 


Total Reading (grade 


3) 


18 
















Or igina 1 procedure 






1 


.60 


4 


33 


1 


. 56 


Unknown 


Modified procedure 






0 


.94 


5 


.04 


0 


.79 


Unknown 


Total Math (grade 5) 




22 
















Or igina 1 procedure 






7 


.14 


5 


67 


5 


.91 


Yes 


Modified procedure 






3 


.15 


5. 


04 


2 


.93 


Yes 


Total Reading (grade 


5) 


19 • 
















Or igina 1 procedure 






1 


47 


4. 


13 


1 


.55 


Unknown 


Modified procedure 






2 


.05 


4. 


70 


1 


90 


Unknown 


Total Math (grade 6) 




27 
















Origina 1 procedure 






1 


11 


6. 


77 


0 


85 


Unknown 


Modified procedure 






1 


06 


6. 


70 


0 


82 


Unknown 



For original procedure, SDpost-pre^ modified procedure, 

^^pos<:-post • 



For three of the four sets of data shown in Table 3-23, rhe numerator 
of the t statistic, gain over expected gain, is lower under the modified 
procedure than under the original procedure; the denominator tends to be 
the same under either procedure. As a result, the t values tend to be 
smaller under the modified procedure* In all of these cases, the conclu- 
sion regarding normal growth would have been the same under either pro- 
cedure. However, in situations where normal growth is only narrowly 
achieved under the original procedure, it may not be achieved under the 
modified procedure. 
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3.8.3 .Stirin^ency of the Criteria 



Ideally, the criteria of normal growth and of educationally signifi- 
cant growth should be established and implemented so that the degree of 
difficulty in attaining the criteria is independent of factors such as 
grade or pretest score. A procedure by which it is easier to demonstrate 
program effectiveness at one grade level than at another would be unfair. 

The stringency or the difficult.y of meeting a criterion can be judged 
both in educational and statistical terms. Educationally, the difficulty 
of meeting the norm-referenced criteria is related to the effort necessary 
for achieving the necessary gains, where effort is measured in terms of 
educational resources and time needed for students to acquire the neces- 
sary skills. While the standard score metric is purported to be an equal 
interval scale of achievement, it is probably not the case that a speci- 
fied gain in standard score requires the same amount of effort inde^^endent 
of grade level or initial standard score. Also, as was emphasized ear- 
lier, effort per se is not enough; it must be effort directed to the 
acquisition of skills measured by the MAT. 

The equal percentile assumption for normal growth demonstrates the 
differences in standard score gains across grade levels necessary for 
normal growth. For Total Reading, a gain of between 7 and 9 standard 
score points is necessary to maintain normal growth at the second grade 
between fall and spring. This decreases to a gain of between 1 and 4 
standard score points for eighth grade. In a few instances, in the upper 
grade levels, the equal percentile assumption dictates that zero gain be- 
tween fall and spring is sufficient for normal growth. Below the 50th 
[.careen tile , the specified standard score gains necessary for normal growth 
art^ quite uniform across percentile ranks within grade. 

The substantial differences across grade may be attributed to any of 
a number of factors: 

• The test items become increasingly irrelevant to the type.s 
of skills being taught in the upper grades. 

• Students tend to reacli an asymptote in their acquisition 
of reading skills, and additional gains require much more 
effort than at the lower grade levels. 

• Less time and effort are spent in the upper grades in ac- 
quiring th.: skills nt^asi/fed by th.e MiVT. 

• Tlu. standardisation pro-jedure v^as defective. 
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If it is assumed that the standardization program produced valid 
norms, then either students appear to reach some asymptote in the upper 
grade levels on reading achievement or the curriculum at the upper grade 
level is irrelevant to the skills Tiaasured by . the MAT. If the curriculum 
is irrelevant;, it may not be the case that the observed low gains in 
standard scores indicate a great degree of difficulty in achieving growth. 
It may merely mean that students are not spending much time learning 
skills relevant to items on the MAT. 

The difficulty of achieving specified gains on the MAT cannot be 
assessed with any degree of accuracy, given current educational theory. 
However, the gains expected under the normal growth assumption may be 
taken as a baseline for assessing the stringency of the criterion of 
educational significance. That is, the gains by the MAT standardization 
group may be taken as representative of the output of programs with an 
average effort expended at each grade level. 

This was the point of view adopted in Section 2.3. It was found n 
there that, since the rate of change in the standard deviation of stan- 
dard scores was small, the one-third norm standard deviation criterion 
for growth is constant over grade levels. Therefore, with the decrease 
in the expected normal growth across grade levels, it appears that the 
criterion of educationally significant growth becomes increasingly strin- 
gent as grade level increases. That is, greater effort seems necessary 
at the upper grade levels, relative to the effort normally expended at 
those levels, to obtain gains that meet the criterion of educational sig- 
nificance. The one-third standard deviation criterion for educationally 
significant growth wa s proposed by Hor s t , Ta 1 Imadge j and K^*" jd me rely as 
a rule of thumb. The results of the study of stringency reinforce this 
poinc of view: No empirical or theoretical basis exists for selecting 
tl.e one-third criterion, and at least from one point of view this cri- 
terion becomes increasingly difficult to meet as a function of grade 
level . 

Statistically, the issue of stringency appears to be more cle^.i - /.-.i r., 
and one factor--the number of students in the analysis--appears to Ij 
more critical than the grade level or pretest score.'-. From a statistical 
point of view, the stringency of the criterion may be expressed in terms 
of the power function that describes the chance of meeting the criterion 
given particular gains (under the ideal conditions underlying the appli- 
cation of the t test). For tho nr.-ref ^'irenced procedure suggeste^l by 
Horst, Tallmadg-, and Wood, rhe power function may be expressed as: 

P / T t I 6 \ , 

y 0.025, N-li / ' 
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where T is thfj test statistic described in Section 2; tQ^Q25^ is thu 

0.025 critical point of a student^s t distribution, with N-1 degrees of 
freedom; and 6 is the noncentrality parameter expressed as Ajn/a^ where 
A is the true difference between the observed and expected spring stan- 
dard scores and ^ is the standard deviation of these differences. Fig- 
ure 3-4 shows the power curves for a few selected sample sizes. Note 
that, in all cases, the probability of passing the normal growth c "iterion- 
given that A is zero--is 0.025. That is, if the "population" gains in 
standard score are exactly what is expected under the normal growth as- 
sumption, the normal growth criterion will be passed in only 25 out of 
1000 replications. Obviously, for a given value of A/a, the chances of 
meeting the criterion increase as the number of students in the analysis 
increases. For example, for A/o = 0.3, the chance of meeting the crite- 
rion is greater than 8 out of 10 when the number of students is about 100, 
drops to less than 4 out of 10 when the number of students is about 30; 
and is less than 2 out of 10 when the number of students is about 10. 

From a statistical point of view, it is plausible and reasonable, 
of course, to have a more stringent requirement for norma"! growth as the 
sample size decreases. However, in most field evaluations the number of 
children in the program to be evaluated is not under the control of the 
evaluators. Therefore, the stringency of the criterion depends to some 
extent on extraneous factors such as the number of sites where the pro- 
gram was implemented, the number of children at the sites who qualified 
for the program, and the optimum number of chlldre.i that could be accom- 
modated in the operational design of the program. For example, some PIPs 
have as few as 4 students at a given grade level and others have as many 
as 779. Other things being equal, including the actual impact of a pro- 
gram on achievement, the program with the larger number of children has 
a much greater chance of demonstrating its effectiveness. 

3.8.4 Modif ica t ions of the Norm-Referenced Procedure 

In the preced Iny; discussion, potent ial weaknesses of the norm- 
re ferenced procedure were described. These include: 

♦ On some tests, th; cxpocLod oosttest sLandaiU scoiu i-i^^.Ml 
'^ii the equal percentile assumption is too low for sLi.idents 
with extremely low pretest scores. 

• There are i nd ica t ions that the expected post test standard 
scores are too high for disadvantaged students, especially 
disadvantaged minority students. 
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• The criterion for educationally significant growth may not 
be of uniform stringency across grade levels. 

• The stringency of the criteria for normal growth and for 
educational] V significant growth depend on the number of 
s t udeti t s in the eva 1 ua t ion . 

This list dues not touch upon some of the more basic criticisms of the 
norm-referenced procedure that were discussed in Section 2. If an eval- 
uator agrees that the conceptual basis for the norm-referenced analysis 
is extremely weak, no minor mod i f ica : ions of the procedure will be sat- 
is f ac tory . 

The modifications necessary for satisfying the first two points 
above are relatively straightforward: Raise the expected posttest stan- 
dard score for ex::remeiy low pretest scores; otherwise, Icwr the ex- 
pected spring standard score. Furthermore, use different expected post- 
test scores for students of different minority status. It is not at all 
clear what modifications of the current procedure can be made to make 
stringency uniform across grades. For one thing, there is no educational 
theory to quantify the effort necessary for achieving a specified gain. 
Therefore, assessing the stringency of the normal growth criterion is 
nor possible. If the normal growth criterion is taken as a baseline, 
under the assumption that it represents the results of an average level 
of effort, it is not clear how educationally significant growth should 
relate to normal growth. For example, if the standard score gain neces- 
sary for .exhibiting educationally significant growth is taken as a con- 
stant multiple of the standard score gain necessary for normal growth, 
it is not at all clear what multiple should be used. The current pro- 
cedurL' provides no guidarce because iho one-third standard deviation 
criterion w^is offered as a rule of thumb with no educational basis for 
its adoption. Under these circumstances, no modification of the educa- 
tionally sik',nif icant growth criterion was examined. 

The dependence of the' stringency of the criterion on the number of 
students m the evaluation is also a factor that is not amenable to a 
modification of the proc-dure after the data have been collected. Theo- 
reti:.-!l3, p:irt the J.si-:i of the evaluation should be a specification 
of the number of children to b^> included, where this number is based on 
considerat L.:)ris of the power of the statistical procedure. When the num- 
ber of children included in evaluation is deteimined in a haphazard 
fashion, there does not appear to be any post hoc formal procedure for 
making the statistical, analysis independent ol the number of children. 

In this c.nic 1 'id in-; secti^>n, then, we will examine the impact on the 
no:-m.r..-ferrric,.d .>na lysis ot eluinges in the specification of noniKil growth 
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for fourth grade Total Reading. Fourth grade was selected because it 
was the only grade in which the sam^f battery level of the >\AT had been 
administered to the MAT standardization ^roup, the compensatory reading 
evaluation group, and the PIP evaluation group. The CR/SL group was used 
to derive the function describing the expected posttest score, given the 
pretest score. A separate function was derived for each of three ethnic/ 
racial groups; whites, bl-acks^, and Spanish surnaxie . A regression anal- 
ysis was used to derive the Ciinction in a pretest standard score range 
between 46 and 88, corresponding to a percentile range between 6 and 94. 
The summary statistics are presented in Table 3-24. 



Table 3-24 

SUMMARY STATISTICS FOR ESTIMATION OF EXPECTED 
POSTTEST TOTAL READING STANDARD SCORE, 
GIVEN PRETEST STANDARD SCO'<E 



Race/Ethnicity 


Number 
of 

Students 


Intercept 


S lopo 


r2 


SD 


1 

SE 
Slop- 


Wh Lte 


1238 


3.32 


1.04 


0.69 


6.2 


0.02 


Black 


619 


-1.60 


1.11 


0.57 


5.8 


0.04 


Spanish surname 


179 


8.31 


0.93 


0.63 


5.8 


•J. 05 



Note: Statistics were derived from data provided by ETS. 



It was felt that the functions still over-predicted posttest stan- 
dard scores because the time interval between pre- and post-test may have 
been as much as a month longer than the interval for the PIP evaluation. 
Also, these children had been in some form of compensatory reading pro- 
gram. It is not clear whethe ' children who were in the PIP program would 
have been in other compensator;, programs if the PIPs had not been imple- 
mented. If the children had been in other compensatory programs, such 
as those funded under Title I, the CR/SL group would appear to be a ^^ood 
comparison group. Otherwise, some adjustment to the ^'lunctions would he 
necess.iry to account for tlie gains attributable to the compensii tory read- 
ing program. For current purposes, each intercept was decreased by 1.5 
standard score points to represent the potential effect of about a month's 
increase in tlie time interval and the potential effect of the compensatory 
reading program. Pelavin and Barker (1976) proposed a rate of growth of 
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about 0.73 standard score poinis per month as a rule of thumb in their 
study of the MAT. Most studies of compensatory education programs have 
found only small effects attributable to the program. Therefore, a cu- 
mulative effect of cibout 1.5 standard score points appears to be reason- 
able for this study. 

For students belov. the 6th percentile lower bounds a constant ex- 
pected spring score was postulated for each rac ia 1 /ethnic group. Prior 
resul ts (see Figure 3-1) had indica ted tha t the expected spr ing score 
for a student witli fall standard scores below the 6th percentile was ap- 
proximately independent of the particular pretest score. The expected 
posttest score was found to be approximately equal to the regression, 
lines evaluated at a pretest standard score of 45. A similar procedure 
was used for students with scores about the 96th percentile. 

The student was u^ed as the unit of analysis. Each student's fall 
scrrc. was used to derive an expected spring score, E. The difference, 
D = S - E, between the observed spring standard scores., and the ex- 
pected spr In St: ores, E, was then calculated. These differences were 
used to calculate the t statistic: 

SD/in 

where 

D = moan d i f EerunC'j 
SD = s ^:aadjrd dev ia t ion of the d if f erences 
n = nu T^b e r oi s t ud c n t s . 

'Jiult-r the .is suii;pt icui that the deviations between observed and ex- 
pc^uLed scor^.-s are approximately independent and normally distiibuted, T 
wili iutVc: a st'.idunt's t distribution with n-1 degrees of freedom. 

s.nuo crLtcricii ah that used in the unmodified analysis was used 
t':St tor ncniKil growth: if the 9 5% confidence interval on the mean 
CvjLdll> al.''OVL zero, the normal growth criterion was said to be 
a L - i i ; it' ti.e ccn i idcncj interval was totally below zero, normal growth 
w<rj sjivl lo bt- rtLL.iined; otherwise, it was assumed unknown whet lie r or 

iK> L :; V- V'l.a .. : j'.v L : i lu; d 'x- n a t t:a inu d . 

j-^r- si', tViL rcsi.lcs of th^- original and modified an;i lysis 
V -.^p'-.^ ... .vj:*:i..'[ ;^r.v;Lh criterion for fourth grade To la I Reading. 

.■.'."ciMii, J L" i 1 L . Il . v • '1 c 1 i ^ I u 11 s Is evldeHt, One project. Con- 

quv:.;, cti-si'^-. : ^.ikriowii" to "yes" relative to meeting the normal 
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Table 3-25 



iSULTS OF ORIGINAL AND MODIFIED NORM-REFERENCED 
PR HIRE FOR FOURTH GRADE TOTAL READING, BY PIP AND SITE 









Original Analysis 


Modified Analysis 




:.ber 


Gain 


Gain 




Meets 


Gain 




Meets 




of 


Over 


Over 


t 


Normal 


Over 


t 


Normal 


PIP/Site 


Students 


Fall* 


Expected* 


Test 


Growth 


Expected 


Test 


Growth 


Catch-up 


83 


2.96 


-2.04 


-3.46 


No 


-1.48 


-2.81 


No 


Blooniington 




2.73 


-2.28 


-2.70 


No 


-1.75 


-2.23 


No 


Brookport 


8 


3.75 


-1.25 


-0.75 


Unknown 


0.01 


0.01 


Unknown 


Gala.t 


1 


4.43 


0.43 


0.11 


Unknown 


■2.49 


-0.95 


Unknown 


Providence Forge 


20 


3.05 


-1.95 


-2.08 


Unknown 


-0.94 


-0.91 


Unknown 


Wayne City 


8 


1.88 


■2.25 


-2.64 


No 


-2.11 


-2.36 


Unknown 


Conquest 


108 


5.56 


1.08 


1.98 


Unknown 


1.24 


2.46 


Yes 


Benton Harbor 


28 


7.36 


3.19 


2.75 


Yes 


3.45 


3.53 


Yes 


Celevland 


53 


4.77 


0.49 


0.65 


Unknown 


0.80 


1.17 


Unknown 


Gloversville 


27 


5.22 


0.22 


0.22 


Unknown 


-0.18 


-0.17 


Unknown 


IRIT 


34 


3.44 


-1.23 


■1.39 


Unknown 


-1.18 


■1.58 


Unknown 


Bloo:nington 


28 


4.39 


0.12 


0.12 


Unknown 


■0.46 


-0.56 


Unknown 


Schenectady 


6 


-1. 00 


-6.00 


-6.71 


No 


■4:56 


-4.94 


No 



* 

In standard score units. 



growth assumption. Only one site, Wayne City, changed from "no»» to '*un- 
known." Under the modified procedure, the gains over expected increased 
for the PIPs shown in the table. At a few siteS; the gains over expected 
decreased because of the increase in expected spring scores for students 
with extremely low fall scores. The difference between the gains of stu- 
dents with extremely low pretest scores and students with higher scores 
is revealing: For the 44 pupils who scored below the 6th percentile in 
the fall, the average gain in standard score was 8.4 points; for the 181 
pupils who scored at the 6th percouile or above, the average gain was 
only 3.3 standarr* score points. 

ba od on the above results, it would appo.ir that modest modifications 
of the norm-referenced procedure will not le-,: ■ t; :> different conclusions 
regarding the effectiveness of the PIPs, excepL in marginal cases. In 
the remaining sections, results of the curriculum-referenced analysis 
are reported. 
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4 FIELD EVALUATION OF IMPLEMENTATION 



4. I Introduction 

Results of field activities undertaken to satisfy our principle of 
description are discussed in this section of the report. In accordance 
with the assumption that, all other things being present, MAT items will 
be learned from an appropriate curriculum, given a reasonable Ir.ngth of 
exposure with a competent teacher, we attt:mpted to uncover exactly what 
the PIP-specif ied instructional techniques and curricula were. We also 
desired to assess the impact that the degree of implementation of t' in- 
structional component had on test scores. 

In this section we discuss the PIP instructional component and how 
we assessed the degree to which it was implemented. Because of the 
PIPs* emphasis on management, last year we examined* the management 
features of the PIP projects. This year we were more interested in the 
hardware/software component of the PIPs and in the classroom instructional 
procedures, which were scattered throughout the packages. 

We immediately discovered that very little was specified relative 
to project instructional style. The PIPs assumed that qualified project 
staff already know how to teach in the style required. The PIPs also em- 
phasized the use of widely available published materials. The immediate 
implication was that the PIPs would not necessarily place an innovative 
instructional process into a conventional atmosphere. At this early 
stage of our investigation, for all we knew, the PIPs' curricula and 
styles would be substantially the same as those they were intended to 
supplement, or in the case of R-3, to replace. In any event, the PIPs' 
reliance on published materials (except for R-3) made it quite likely 
that, insofar as test scores were concerned, PIP effects and school ef- 
fects would be the same. It would be just as reasonable to attribute 
gains to the regular school as to the PIP-induced add-on, if the ^wo 
teaching styles and curricula were substantially the same. 



The IRIT and Catch-Up PIPs warn that their projects must be made 
distinct from the regular curriculum, " 
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We set up our site visit procedures to accomplisli two objectives. 
The first was to assess implementation and the factors that influenced 
It, which would all- us to make reasoned policy recommendations about 
packaging. A good assessment of implementation would free us from com- 
plete dependence on the norm-referenced analysis for evidence of project 
success. If we could show that test scores increased with good implemen- 
tation, would have evidence consistent with PIP effectiveness. 

The second objective for our fieldwork was to collect information 
on the curriculum that the project teacher actually used» These data 
would make it possible to compare PIP-specif ied curricula wicb the MAT 
items and to compare MAT items with the items in the tests used for Dis- 
semination and Review Panel approval. 

We had some success in achieving both objectives. Section 5 pre- 
sents results of our curriculum analysis and describes the data we have 
for comparing the MAT with the original Dissemination and Review Panel 
test. Section 6 contains formal analyses that use our data on the cur- 
riculum and on implementation to assess achievement impacts. 

Section 4.2 describes our method of approach for recording on-site 
observations. 

4,2 Methodology for On-Site Observations 

An evaluation that attempts to assess the worth of an innovation 
without systematically describing the innovation, as realized in the 
field, is in danger of evaluating the wrong program. The program in the 
field is, after all, the presumed causal agent for any outccies of 
interest . 

The methodological problem is how an observer may correctly de- 
scribe what is in the field and hov; he communicates this to the inter- 
ested audience. Broadly speaking, we distinguished two paths as being 
available to us. The first, we call the objective-subjective, or hard- 
nosed approach. T second, we call the subjective-objective, or the 
soft-headed appro 

In the objective-subjective approach, the observer's notebooks are 
forms in which he records the presence of objective factors, such as the 
lighting and seating patterns in a classroom. There may be space to re- 
cord some fairly molar behaviors, such as a teacher asking questions. 
Characteristically, in the hard-nosed method data are recorded at a 
fairly fine lovol of detail, scored, and then statistically aggregated 
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into measures of various factors. The analyst may even construct a vari- 
able called ''implementation. 

In the subjective-objective approach, the observer takes his notes 
in ordinary language and reports them novelistically • The soft-headed 
approach is characteristically conducted at a molar level, with judg- 
ments and conclusions being stated and justified in text. 

We call the hard-nosed approach objective-subjective because it is 
fairly objective during data collection but subjective at analysis time, 
since the choice of metric is subjective. On the other hand, we call 
the soft-headed approach subjective-objective because at data collection 
tim? the observer makes judgments, but at analysis time the analyst can 
only treat the judgment as objective. Characteristically, the objective- 
subjective approach ends in the analysis of a real-valued implementation 
scale, while the subjective-objective method ends in the analysis of the 
yes/no judgment of implementation. Much can be said for, and against, 
each approach. Last year we relied principally on the objective- 
subjective technique. This year we relied principally on the subjective- 
objective technique. 

The first- and second-year observation methodologies are related in 
that, in both years, we used forms as the primary components of the ob- 
servers' notebooks and began the second-year site visit by using the 
first year's haphazardly selected sample of schools and teachers. How- 
ever, our intention was not to repeat last year's general assessment^of 
implementation. Rather, we wished to focus in depth on the extremes of 
implementation; we used the information from the first year to select 
those extremes. Ultimately we hoped to determine whether the successful 
implementation of the PIP instructional program made a difference in the 
children's achievement scores. 

Table 4-1 shows the dates of the site visits, by project, and the 
site visitor responsible for each. Generally the visits were made after 
fall testing and before spring testing. Only two PIPs had more than one 
site visitor. Because it was not economically possible to send more 
than one visitor to each site, the validity of our assessments of 
teacher implementation is principally dependent on the judgment and ob- 
jectivity of a single site visitor. Although undetected errors are pos- 
sible, we feel that we have been successful in comprehending each PIP' s 
intent. 

The steps we took to define the intent of each PIP and the resultant 
interpretation of intent are described below. 
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Table 4-1 



SITE VISITS FOR OBSERVATION AND INTERVIEWS 





Visit 


Date 




Project 


Fall 1975 


Spring 1976 


Site Visitor 


Catch-Up 








Blootnington 


12/9-12/12 


2/18-2/19 


Margaret Needels 


Brookpor t 


12/4-12/5 


3/11-3/12 


Phil Giesen 


Galax 




J/ iO-J/ i7 


PVi "il f^ipQpn 




12/9-12/10 


3/15-3/16 


Phil Giesen 


Wayne City 


12/3-12/4 


3/9-3/10 


Phil Giesen 


Conquest 










12/12-12/15 


3/11-3/15 


Casse Duarte 


L# leve iciua 


12/8-12/10 


3/8-3/lU 


Casse Duarte 


Gloversville 


12/16-12/18 


3/16-3/19 


Casse Duarte 


HIT 








JLeX J-llg Lull 


12/7-12/9 


3/10-3/12 


Dorothy Booth 


Olean 


12/2-12/5 


3/17-3/19 


Dorothy Boo 


IRIT 








D J QOlTl in.^ I. Ulf 


12/11-12/12 


2/16-2/17 


Margaret Needels 




12/15-12/16 


2/9-2/10 


Margaret Needels 




N n V i ^ 1 

11 V_/ V ^ O -A- ^ 


2/12-2/13 


Margaret Needels 


PTR 








Canton 


12/1-12/2 


4/12-4/15 


Jay Cross 


Dallas 


12/3-12/5 


3/31, 4/1, 4/7 


Jay Cross 


Charlotte 


12/10-12/12 


3/17-3/19 


Georgia Gillis 


Lake Village 


12/11-12/12 


3/22-3/24 


Dorothy Booth 


Lorain 


12/5-12/9 


4/27-4/29 


Georgia Gillis 


Schenectady 


12/2-12/4 


3/9-3/11 


Georgia Gillis 



No site visit because of teachers' strike at this site. 
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4.3 Identification of PIP Instructional Componerita 

4.3-1 Results of the PIP Washington. D .C«. Co nference 

At ths end of the first year's study^ we were aware that most of 
the tryout projects had been generally successful in implementing the ex- 
plicit PIP specifications. These were the management specifications and 
those areas of the instructional program that were clear enough to make 
verification possible. However, problems at each project caused dissimi- 
larities across sites within the same PIP. With the increased emphasis 
on achievement gains in the second year; it was important that the proj- 
ects be given an opportunity to resolve problems caused by attempts to 
meet certain specifications literally and to discuss whether they were 
obliged to comply with specifications in cases where they had already es- 
tablished nonreplications. Thus, in September 1975, U50E held a two-day 
conference in Washington for project directors and selected teachers and 
administrators from each project c:\te. Representatives from the devel- 
oper cites, KMC, and SRI also attended. Because the purpose of the con- 
ference was to promote unifomity in interpretation and procedure among 
the tryout sites within each PIP, project directors and teachers ex- 
changed experiences and, with the help of the originating site director, 
agrees" on resolutions of some of the problems they had encountered in 
trying to follov; the PIP specifications. Records were kept to provide 
early iuf ormati.-^n for PIP revisions. 

Following the conference, summaries of the agreed-upon interpreta- 
tions, revisions, and clarifications for each PIP were printed and dis- 
tributed to the project directors. The complete summaries are repro- 
duced in Appendix C. The following examples show the substance of the 
changes. 

• Catch-Up - -Criterion-referenced tests were interpreted as 
teaching tools to be used about once a week so that teach- 
ers could keep abreast of an individual student's needs. 
Teachers were cautioned that the use of Catch-Up materials 
in studeats* regular classrooms would minimize the materi- 
als' effectiveness in the project; yat, students* under- 
standing would be increased by the use of project materi- 
als that correlate with those used in the regular 
classroom. 

• Conquest — The role of the supervising clinician in the 
Conquest program was clarified by setting down some spec- 
ifications for that person's duties, hours, and teaching 
assignments. Specifically, the supervising clinician's 
main duties were to assist the project director with 
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training and administration, to monitoi" and asi;isf: clini- 
cians, and to teach some students, wicK tlnL amount of rime 
spent on each of these duties to vary with the size of the 
project and the number of clinicians to be ob'jorved^ 

/\ revised diagnostic tvst battery was issued to partici- 
pants v'ith a warr.in*^ that the 12 tests must be adminis- 
tered in the order of listing. However, it was agreed 
that the lia^raoscic process itself is more important than 
the teuts used and that some substitutes might be ac- 
ceptable . 

HIT --Many clarif i -^viions of the instructional ^-^rocedures 
resulted from the HIT meetings. Drill in math uses many 
different materials, commercial or teacher-made, designed 
to teach basic facts; there should be sufficient materials 
to keep drill from being unnecessarily monotonous. Drill 
in reading is aimed at fluent word recognition, which 
should not be interrupted by having students spell or 
write words letter by letter. Drill should be varied by 
occasionally making a game or contest out of drill words. 

IRIT --IRIT participants clarified some instructional and 
management specifications: Basal readers that are incor- 
porated in the program at the request of sending teachers 
should supplement but not replace core materials; learr.ing 
machines could be used for practice or as motivators but 
should not be given excessive emphasis; IRIT teachers 
should give students end-of-cycle placement tests and ad- 
vise sending teachers of che results; teachers should spe- 
cialize in one area of instruction rather than rotate 
among areas during a given year. 

PTR--Tryout project directors insisted that materials be 
included in the PIP for use with students who do not have 
reading readiness skills. The Alphabet Skills Book, with 
which they had achieved some success, was recommended for 
this purpose. The originating site's project director re- 
ported that the basal reading series does not have to 
match the tutoring kits used in PTR and urged that future 
PIP projects be allowed to adapt their programs to local 
conditions. All participants agreed that future projects 
should have access to the experienced project directors 
for technical assistance with management and instructional 
problems . 
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• R-2""The R-3 participants were reminded that gaming/ 

simulations (G/S) and contracts were integral parts of the 
program and were urged to use them more frequently. Sum- 
mer workshops were recommended to enable teachers to adapt 
the G/S activities for easier integration into the cur- 
riculum. 

As a result of the Washington conference, all participants had a 
hettdv idea of how to implement some parts of their programs and of the 
importance to the evaluation of replicating the PIP guidelines. There 
were better understanding and greater consensus about which PIP specifi- 
cations were rigid and which flexible, and about which were of very high 
priority and which of lower priority in claiming replication. 

In our efforts to systematize the description of the PIP instruc- 
tional programs, we found it necessary to provide a framework from which 
we could view all of the specified classroom processes. Following the 
Washington conference, detailed program descriptions for each PIP were 
written to draw together suggestions and specifications from those sec- 
tions of the PIP components that bore on instruction and to incorporate 
the revisions and clarifications formulated at the conference. The fol- 
lowing section reports the results of .our efforts. We regard these de- 
scriptions as the criteria! interpretation of the PIPs' instructional 
components as modified by the Washington conference. 

^•3.2 PIP Instructional Program Descriptions 

The instructional programs were not presented intact in the PIPs; 
rather, some elements of instruction were described briefly in eight of 
uhe nine components of each PIP. Since we desired to know how all of 
these pieces fit into a coherent approach, we wrote a description of 
each pip's instructional program. This effort entailed pulling out the 
appropriate sections from the PIPs and incorporating the revisions and 
clarifications that resulted from the discussions at the Washington con- 
ference . 

While the descriptions were being written, it became clear that we 
were flavoring them with our own interpretations of what the guidelines 
should convey, because our understanding of the programs had changed 
since onv first reading of the PIPs. Our visits to some of the origi- 
nating sites, our discussions of interpretations throughout the first 
year with the RMC special:, ts and with the project staff at the tryout 
sites, and our own educational philosophy had all contributed to a 
clearer idea of what each PIP program should be. The revised program 
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descriptions for each PIP follow. It is against these descriptions th^t 
we judged implementation and the quality of teaching at each project site. 

4.3-2.1 Catch-U p Instructional Program 

Project Catch-Up is designed to help children who are in the 
lower quartiles of their -Tge-mates in reading and math skills. It is 
not intended^ however., to include children who have serious emotional 
problems or who suffer mental retardation. The idea is to focus atten- 
tion on those children who^ somewhere along the line, have failed to 
learn the normal content materials (whether prereading^ math skills, or 
materials with more advanced content). 

The causes for these deficiencies in content knowledge were 
uevcr analyzed in depth at the original site, but were assumed to be 
typical causes — for example, less than sufficient attention at home, 
lack of exposure to a stimulating enA'ironment, or, perhaps, language 
difficulties. 

The primary philosophic belief underlying the Catch-Up style 
is that any normal child can learn at the same rate (as his age-mates) 
as long as this belief is conveyed to the child and that he will achieve 
success through exposure to a planned instructional program. The key to 
the program^ therefore, is a coordinated effort that focuses on specific 
needs of the child and is carried out in a positive "you can learn'* at- 
mosphere. The essential elements of the PIP instructional p . .■ram are: 

• Identification of the specific content deficiencies - 

• Assignrrzrrit of relevant materials. 

• Maintenance of a positive atmosphere^ both by assuring 
success and by giving positive feedback. 

• Maintenanr^e of the child ' s enthusiasm. 

Specific content deficiencies in reading are identified pri- 
marily through the use of criterion-referenced evaluation materials, spe- 
cifically Random House. Problem areas^ as well as the component parts 
of these areas, are identified through this criterion-referenced series, 
and skill needs'^are identified in detail. For example^ iden ti fying fi- 
nal single consonar.t sounds'' and "identifying vowels modified by r," are 
skills that would be examined. 
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The mean^ for identification, of problem areas in mathemacics 
are less well defineci ? y the PIP* The main vehicle appears to be the 
placement exam in the Sullivan Math Program, but this ^.xar" does not re- 
fer to any specific technique or system for identifying weaknesses in math 



In both reading and math these diagnostic procedures are sup- 
ported by a variety of other methods: 

<♦ Constant communication between the lab instructor and 
the child* s regular teacher to secure information 
about weaknesses identified in the regular classroom. 

• Constc-nt r ; coring of the child through periodic 
quizzes and review exams. 

• Contact with parents for insight into the needs of 
the child. 

• Constant observation. 

• Securing the confidence of the child to enable him to 
express his own needs. 



Instructors should work with a small number r.-f children and 
should be assigned complete responsibility for the children's progress 
in both reading and math. The small groups, along with careful record 
keeping, enable the instructor to develop and maintain an in-depth under- 
standing of each child's needs. 

The assignment of materials that are relevant to the exact 
needs of the child is possible once specific weaknesses have been identi- 
fied. Materials must be carefully assigned because the constant student 
success that Catch-Up tries to foster can be attained only if materials 
are geared to the individual child. Reading assignments are primarily 
keyed by tVu criterion-referenced materials, since they provide an index 
of some ma.crials designed to attack specific deficiencies. 



A list oi additional relevant materials developed by instruc- 
tors L-c the original site augments those indexed by Random House and is 
maintained by a card index system keyed to the criterion materials. 

The positive, success-oriented techniques come into play after 
needs have been identified and relevant fterials located. Careful 
planning assures that the child can ach. ^ constant success and there- 
fore a positive view of his own capabilities. The following procedures 
ensure the positive atmosphere of the program: 
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• The materials should be relevant, success oriented, 
and of a largo variety. 

» The difficulty and number of assignments should be 
btisv3d on the current learni. g style of the child and 
sl^Duld be increased only as indicated by the c. Id's 
progress. 

• Constant monitoring and observation of the child are 
required for noting any significant changes in the 
child* s netds. 

• Clear a..d complete written progress reports should be 
maintained for constant reference . 

• The teacher should be very positive and should re- 
frain from negative comments such as "You're wrong." 

• The teacher should "join in" with the students on 
games or discussions and thereby show the student 
that he can beat even the teacher on some things. 

• The instructor should know the student so well that 
he can help in noninstructional areas, which also in- 
fluence the child's self-image. The PIP specifically 
mentions becoming a friend and helping with regular 
classwork, resolving conflicts, and attempting lo 
convey the child's needs to the parent. 

• All activities should be carried out in a lab envi- 
ronment that is designed to add to the positive atmo- 
sphere. The lab should be bright and colorful with 
many displays, including the children's work and eth- 
nic themes - 

• The children should be encouraged to become partners 
in the education endeavor by being allowed to under- 
stand why specific assignments are made each day. 

• The PIP also states that the instructor should be 
free to choose any instructional "approach,'^ but this 
should probably read instructional "materials , " since 
materials can be used in any way the instructor deems 
practical. Experimentation should be ..encouraged, and 
each teacher should have his own funds for purchasing 
materials he likes. 

All of this instructional activity i provided with the under- 
standing that, unless the child is interested and enthusiastic about 
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what he is doing, he will not be motivated to laarn. The operating pro- 
cedures and the materials selected should be planned with this in mind. 
The main procedures to be followed are: 

• A variety of materials should be available and uti- 
lized. Some materials intrigue some children, bat 
not all. The instructor should use the materials 
that are. right for the given child. 

^ • Teaching machines should be used, since come children 

are intrigued with the novelty and as a result begin 
to like their work. 

• Games should be available because they add to the ex- 
citement of learning. 

• Effort should be made to avoid any overlap with mate- 
rials used in the regular classroom. This maintains 
the freshness of the lab and prevents the child from 
feeling that the lab is just an extension of the 
classroom. 

• Some free choice of activities and materials should 
be offered so that the child may express his prefer- 
ences. Given the careful planning and specificity of 
assignments, free choice of materials is possible 
only when assignments have been completed or when a 
choice exists between relevant materials. Some lim- 
ited time may also be set aside for free reading or 
extra time at the machines if it does not interfere 
with the schedules of others. 

• Children should be allowed some privacy at timet '-for 
exaimple in portable carrels or a somewhat isolated, 
quiet, reading-learning center. 

Only by employing the careful procedures out?,ined above and carrying 
*:hem out in a positive atmosphere could the original lab attain the 
gains they did. 

4.3.2.2 Conquest Instructional Program 

Project Conquest is a supplemental reading program designed 
from "a comprehensive clinical point of view and aimed at bringing a re- 
mediable student up to grade level in reading. The design of Conquest 
is based on the following premises; 
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• Many students are not progressing in reading at the 
rate possible or to their potential, 

• A pull-out program with a diagnostic-prescriptive ap- 
proach can assess the needs of the student and then 
"cure" them. 

• A reduced student-t Bacher ratio provides ;..n opportu- 
nity for individual attention and an individualized 
program of instruction. 

The instructional treatment for students consists of three 
phases: diagnosis, prescription, and remediation. Students are re- 
ferred to the labs for initial diagnostic screening that incorporates a 

range of diagnostic reading tests, as well as auditory and visual 

it 

screening to assess reading handicaps and needs. A profile is then de- 
veloped for each studei.t; it should contain a graphic and/or numerical 
representation of the student's needs. Those who are one to two years 
behind grade level and show potential for remediation (upgrading in 
reading to grade level or as close as possible) are selected for a year 
of treatment in Conquest. 

The second p iase--prescription--then begins. A procedure for 
smooth transition from diagnosis to remediation was not satisfactorily 
provided by the PIP; however, it states that the child's program should 
be tailored to the child's needs. The PIP allows clinicians to use 
their own professional ingenuity to develop an individual prescription 
for each child's instructional program. 

After prescription, remediation begins. Clinicians work with 
six students, 50 minutes a day, for four or five days a week. Each ses- 
sion, the Conquest student should experience three or four activities in 
the following areas: programmed reading, comprehension, phonics/ 
voc.ibulary/sight words, and oral and/or recreational reading. Instruc- 
tion in at least one of the areas should be assisted through the use of 
a teaching machine. The clinician is directed to personalize attention, 
to motivate, and to provide instructional situations that will ensure 
some success for the student. Extensive planning and record keeping are 
e:<pected of the clinician so as to maximize instruction and minimize 
waste of Cime. Clinicians must keep foluers, daily record sheets. 



Diagnostic screening is considered a continuous process. As students 
gain skills in subareas, they should be rediagnosed for other deficien- 
cies and then ro.mcJlated in those areas* 
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connnercial record sheets, and detailed l./*;son plans for all students. 
Game days held for the stud*' its it the end of each week as a reward for 
working haru should be used to reinforce skills taught in a different 
manner during the week. In <^eneral, the clinician's role during an in- 
structional period should be as follows: 

• The clinician should be -^ble to -nonitor the work of 
all six students, provide individual attention as 
needed, and still work with small groups when appro- 
priate . 

• Whether in a group, or on a one-to-one basis, the 
clinician should observe the behavior of each child 
and teach to the child, rather than to the group. 

Students^ behavior during an instructional period should re- 
flect the planni^ig and teaching style of the teacher. Students should 
be diligent workers, as evidenced by the time spent working on their 
tasks. They shuulc' iter the classroom knowing what is expected of them 
in terms of woik. iney should often work independently in carrels, af- 
ter having builr up self-confidence and the foundation for independent 
study. The student should usually begin the session with programmed 
reading--keeping detailed records of his own progress, and reading his 
daily record '-.beet for assignments in other subject areas. A student 
should move iirom one activity to another with a minimum of wasted time. 

The instruct i.onal program, even within the previously de- 
scribed guidelines, may vury somewhat from lab to lab; however, the ap- 
proach should be consistent among all instructional staff. Students in 
all labs should undergo diagnosis followed bv an individualized instruc- 
tional program that includes a variety of activities, a variety of 
teaching media, and careful record keeping. 

4.3.2.3 High Intensity Tutoring (HIT) Instructional Program 

High Intensity Tutoring (HIT) is designed to raise achievem, it 
levels in reading and math skills for middle school age students who are 
achieving one to five years below grade -level. Tutees are sixth .graders 
and some seventh graders; tutors include eighth graders and some seventh 
graders. Tutees are chosen from candidates who perform farthest below 
grade level and are selected on the basis of spring test scores and 
teacher recommendations. 
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A school participating in the HIT program should have one 
reading center and one math center^ each staffed with one teacher and 
two aides who act primarily as instructional facilitators and book- 
keepers. Tutors for the project should be volunteer students^ who them- 
selves perform at least one year below grade level. 

Tutoring sessions are 30 minutes and are divided into 10- 
minute drill periods and 20-minute workbook sessions. Each tutee at- 
tends one tutoring session four times a week. Ideally,, there should be 
twice as many tutors as tutees so that enough tutors are available for 
each tutoring session. 

Each tutor sits at a desk close to his tutee so that both can 
see the materials being used. Furniture should be placed so that teach- 
ers and aides can circulate freely among the pairs of students. 

Folders of materials should be available to each tutoring pair 
at the beginning of each tutoring session. 'The tutor should always be 
ready to prompt or encour.i-e his tutee^ but should never lecture or over- 
explain. He should use brief questions or corrections instead. The tu- 
tor should keep track of correct and incorrect answers by means of a 
tally sheet, using a slash for a correct answer, a zero for an incorrect 
answer, and an K when he has encouraged or complimented the tutee. Cor- 
rect responses represent earned points to be entered into individual 
bankbooks by the staff daily and redeemed on "paydays'' for rewards. 

Some tutoring sessions should be designated as game days to 
provide a change of pace for both tutees and tutors. 

Tul-.ee.>- and tutors should be motivated in their work by rewards, 
as v/ell as by encouragement and attention to tlieir progress and needs in 
Liu cl.isKrooni. Th.- rewards arc candy for the tvtees and field trips for 
the tutor.^. Paydays, already mentioned should be held once or twice a 
ir.ontb for tutees, so that they may exchange points earned in class for 
the rcw^irds they '.vant. Different rewards have different values. During 
paydavs, tuti'v-s :,-a\ sptMuI whatever points tliey choose, and a new balance 
is entered intu their [)ankbooks. 

Adult .-tafl should circulate among the tutoring pairs, bein;:: 
constantly aware of each tutee' s progress and problems and of the mate- 
rials bein;; covered. Staff sliould always be ready to reinforce a tu- 
tor's aporoach or uive lielp when it is needed. Adult staff n.us:: also be 
aware of l.hu. c!ui;r<.e of rapport h^'tween tutors and tutees and raiow when a 
tiULor is ni.'L doir.-, .■•■11 v;Lth a Lutee. Adult staff should take turn.'.; tvi- 
torin^ i£ tut.' f-.-..' lul')r:\ are i^r-s -it. 
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Staff must record daily each tutce's progress by means of p.n 
individual chart. That is^ if the number of correct answers for a tutee 
fails below 907o for three days in a row, the difficulty of the materials 
is decreased and the student goes to a less difficult section. If the 
number of correct answers rises above the 947o mark for three consecutive 
days, the difficulty of the material is increased. At the end of a sec- 
tion of instruction, a test covering that level of material is given. 
If the student scores 857o or better, he is allowed to begin a new level; 
if not, the tutee must review the material again. 

The materials used for the 10-minute drill are Hegge, Kirk and 
Kirk word lists and math flash cards. Sullivan reading and math work- 
books are used during the last 20 minutes of the tutoring session. 
Teacher-made materials as well as games should be used as alternatives 
for the tutees when they need a change in routine. 

The HIT reading component emphasizes basic phonics; reading 
comprehension is not part of the program. The following words are sam- 
ples from Drill 6 of Hegge, Kirk and Kirk, in which a particular sound 
for the letter u is being learned: hut, run, pup, mug, rub, mud, and 
hum. The words are first shown spaced well apart from each other to fa- 
cilitate reading of them. Gradually the words are placed closer to each 
other. The print is clear and easy to read. Later, in Drill 7 (a re- 
view), words containing the u sound will be interspersed with other 
sounds in words like cat and sip. To pass the review, the tutee must be 
able to read this combination of sounds with ease. 

A Sullivan reading workbook might consist of sections from 
which the tutee reads sentences aloud and answers questions as to the^ 
content of the material. An example of a sentence for which a tutee 
might fill in letters or words is as follows: "A bellboy works in a ho- 
tel. This b_llb_y is taking the woman's luggage to her room." In the 
next series of pictures, different letters or words will be missing. 
Discriminatory responses are also used. In one picture a man dressed in 
an ordinary business suit and another man dressed in a cowboy suit are 
shown, and the sentence underneath asks whether the man on the left or 
on the right is dressed as a cowboy. 

The math flash cards are designed with the probl.:;- and answer 
on one side and the problem without the answer on the other side. 
During the first 10 minutes of the session, the tutee uses the flash 
cards to learn number facts. The tutee may first go through the flash 
cards reading aloud from the side that includes both problem and answer. 
The tutor then holds up the flash cards one at a time showing the side 
with the problem only, and the tutee furnishes Llie answer. In tlie 
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remaining 20 minutes, the tutee works out math problems by hand in the 
Sullivan math workbook. The tutor uses a paper slide to cover and then 
reveal answers as the tutee goes from^one problem to another. 

As the tutee learns the simple skills in reading and math, 
problems and sentence structure become more difficult. Increases in 
level of difficulty are accomplished by skipping pages in the books, and 
decreases are accomplished by reviewing pages. The HIT center teacher 
should review the performance of each tutee daily and make all decisions 
regarding instruction. 

The atmosphere of the HIT center should be enthusiastic and 
task persistent. There should be a-busy muT^nur of voices as tutees say 
their words and number facts aloud. They should give responses at a con- 
sistent pace--fast enough so that interest is maintained, but slow 
enough so that tutors can tally easily after each response. The tutee 
learns by actively practicing the skills he is learning^ not by lis- 
tening to explanations from his tutor or from adults. 



4.3.2,4 Intensive Reading Instructional Teams (IRIT) 
Instructional Program 

The Intensive Reading Instructional Teams (IRIT) project is de- 
sii;aed to raise the achievement level of pupils who are deficient in the 
basic skills o£ language and reading, IRIT also attempts to improve the 
self-image of the students and to develop motivated self-directed learn- 
ers. IRIT teachers guide students toward developing an appreciation for 
and pleasure in reading. 

The IRIT instructional year is divided into three ll-week cy- 
cles. Students are ..^rolled for one of these three cycles, and A5 stu- 
dents are enrolled for each cycle. Instruction is provided each morning,, 
for approximately three hours, 

Thi? IRLT program has three teachers, each with his own class- 
room and each instructr'ng in a different area of reading. The 45 stu- 
dents are divided into tiiree heterogeneous groups of 15 students each. 
After each 50-minut(j period, these groups move from one of the teachers 
to another. Thus each teacher sees all 45 students during the morning, 

llie areas of reading that are taught by the three IRIT teach- 
ers are decoding, vocabulciry-comprehension, and individualized reading. 
Each teacher's rooi;i should be different in materials used and particular 
reading skills l-.aiight. Since students are exposed to an intensive 
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reading program for the entire morning, they should be exposed to vari- 
ability among the three classrooms. There is of course some overlapping 
in skills, but the IRIT teachers should attempt to identify these skills 
and assign each to one of the three classrooms. 

Although each classroom is unique in materials used and skills 
taught, IRIT teachers are a team and a continuity should exist in each 
child's instructional program. To accomplish this, IRIT teachers should 
meet regularly to discuss the progress of each student and to inform 
each other of specific needs of individual, students. 

The key to IRIT is individualization. To individualize is no 
easy task, since teachers have 45 students per cycle for whom they must 
prescribe instruction. Moreover, every 11 weeks a new group of students 
is enrolled in the program. Their -needs must be diagnosed, and the best 
way of helping them must be identified. 

Careful diagnosis of reading problems is the foundation of the 
IRIT instructional process. The first week of each cycle is devoted 
largely to testing so that team teachers will have accurate diagnoses on 
which to base their lesson planning. The diagnostic instruments recom- 
mended in the PIP are the Batel Phonics and the Craft Word Attack and 
Comprehension Tests. An additional test used is the Random House Cri- 
terion Reading. Each team teacher is expected to select and administer 
diagnostic test^; appropriate to her area of specialization. Each may 
also use instruments built into the instructional materials her students 
are using. In still other cases, she may develop her own tests to meet 
specific student needs. Clearly, the IRIT teacher is expected to come 
to the project with a good grasp of the role of diagnostic testing in 
the teaching of reading. 

The teacher must also be familiar with specified comm^.rcial ma- 
terials. After the student's deficieat skill areas have been id ^.ntif ied, 
the teacher assigns those materials that will help the student im^^jove 
the specific skills. IRIT teachers shoul/i noL automatically assign i-ro- 
grammed materials to their students, having them "run" through materials 
designated by the pu^"'.shcrs. Rather, the teachers should use mate- 
rials as tools for i< Jividualization. An IRIT student might be assigned 
two or three different ma -e rials chat inst:vuci: in the same skill, but 
all materials should b*-' reviewed and ic'entified by the teacher as help- 
ful for that particular student. . < 

As well as helping sr.udents l' . those arc-.i.s in which they are 
deficient, IRIT ttiychers must be ire of 'he r-jed to help students grow 
in those arf.'a:s in v.'l'. ich they arc strung. the iRIT teacher might have 
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several materials that the students enjoy and might assign these on a 
regular basis . 

Because of the intense individualization of IRIT, each stt'do)*, c 
must work independently on his assignments. When students enter a cl.. i s- 
room, they should go immediately to their individual folders, which I. c;id 
their assignments for that period. IRIT teachers have a full afternucn 
for planning, and need it, since they must put these daily assignmer r: > 
in 45 folders. Each student should receive two or three assignmen-.s :iov 
one period; however, the number may vary, depending on the materials. 
Some IRIT teachers feel it best to allow students to work for a Ijngor 
period of time vjith certain materials. 

Occasionally, students will work in small groups--for p.<ampl.^, 
when playing a language skill game, or when the teacher has identified 
several students who could benefit from working with her in a smaV 
group. Usually, however, students work independently. The teacher 
should walk around the room monitoring students, stopping to help as 
needed. Students should feel fvi\e i j approach her for help. Every min- 
uxe i:ii>;ortant in the IRIT program, and teachers should see that stu- 
dents da not; siuiply sit, wasting tiina. - 

A 100 lin^^ ot excitement should permeate IRIT classr . The 
students know that tbsir assignments represent individual ar. l ;,ion. 
They also know ^hat they are expected to be independent worl'.e:- -, but 
should always feel free to ask for help T.-hen needed. 

During the first week of the cycle, students must be taught 
how to use the various teaching machine:, and m3::erials available in the 
IRIT cla>srooms. They must also learn the classroom rcul.i.ne and the pro- 
cedure for recordivij?, their own progress in their f olden- on wall 
charts . 

The IRIT Mioruing shoul' be busy for both teachers r--<<^ students. 
The many materials r.nd machines, "r.he three unique c l/.^rsrooms, and three 
different tcach-jro provide enough variety to motivate students to stay 
with the work for the entire thr* e hours. 

A look at the ihree IRIT classrooms reveals the following dis- 
tinctions. The der.oding classroom is perhaps the most oriented toward 
very specific skills. In this classroom, students should be helped with 
any difficulties in phonology .:nd should do a great deal of drill work. 
Here it is important th.ir the decoding teacher use a variety of materi- 
als in new r.nd in teres tiiig wa\ s to maintain student enthusiasm. 
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The emphasis in the vocabulary-comprehension classroom is on 
reading comprehension. Students should be assigned a variety of reading 
materials, often accompanied by work sheets or other assij^nments . The 
Random House Criterion Reading has been most benef i r. :' ::1 in . Jentifying 
skills related to this room. The teacher must gua^ s-.^aL.-ist simply 
scheduling students for the use of various programhu. j Tiurerials; she 
must use the materials in a prescriptive manner. Because activity in 
this room deals with skills that overlap activity in the decoding and in- 
dividualized reading rooms, the team must decide where these skills 
should be assigned for treatment. 

Of the three classrooms, the individualized reading room al- 
lows the most freedom. In this room, the teacher should attempt to moti- 
vate and guide students toward an interest in reading. Students should 
h(\ given freedom to choose books of interest, either from a variety of 
L oks at a specific reading level or from a variety of books at all 
reading levels. A visit this room should show students reading a va- 
riety of books, some of which have accompanying tapes. The teacher 
should have conferences with students (preferably with one student at a 
time) after books are read. Students should mark on a wall chart or in 
their personal folders any work Lhey have completed and indicate if they 
are ready for a conference with the teacher. 

4.3.2.5 Programed Tutorial Reading (PTR) Instructional Program 

The objective of Programed Tutorial Reading (PTR) is to improve 
the reading ability and self-confidence of underachieving first grade 
students. Pa . ev.iluation of PTR programs indicates that, when properly 
implemented, the program has been an effective supplement to conventional 
classroom teaching. Indic^itions are that it has been most effective with 
students who fall in the bottom quartile on national test score distribu- 
tions in reading. 

The instructional setting consists of a tutoring station that 
allows side-by-side seating for the tutor and the tutee. The tutoring 
location may be in a corner of the regular classroom, in a vacant class- 
room, or in any available school space that is free of disturbances. 

Each student is tutiored for 15 minutes each day by the same 
full-time paraprofess Lona 1 tutor for the duration of the school year. 
Duriny, the 15-minute session, the tulor should adhere to tightly designed 
tutor)' x» programs that carefully delineate and control instructional pat- 
terns usod and should limit all decisions about a student's performance 
to judt'.ing the correctness of each response;. Throughout the 15-minuto 
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session, Che tutor records the student's failure for each reading item 
within a lesson on a record sheet so as to determine which items are to 
be presented again on succeeding attempts. In the final analysis, every- 
thing the tutor does and says is determined by what the record sheet 
indicates the student knows or does not know. 

At present, materials used in PTR are available from six pub- 
lishing companies in the form of, tutoring kits: The materials from each 
of the publishers have been designed as supplements to pre-primer and 
basal readers used in regular first grade classrooms. Each tutoring kit 
consists of the basal reader used in the regular classroom, a comprehen- 
sion and word analysis book, word list cards, record sheets, and a tu- 
tor's guide. The tut^ -'s guide specifies teaching procedures in detail. 
It also contains a master list that specifies the order in which tu- 
toring lessons are to be presented. A tutoring kit is needed for each 
tutor . 

The Alphabet Skills booklet, published by Indiana University, 
is recommended as a prelude to the tutorial kit for children with no pre- 
vious reading experience in kindergarten. 

PTR instruction is both methodical and repetitive. It is dic- 
tated by 11 different tutoring programs called Item Programs (e.g., 
sight re^ding, reading, question, completion, and story), which were de- 
veloped to supplement reading skills such as word analysis, comprehen- 
sion, oral reading, and sentence construction. Each program specifies 
in detail what to teach and how to teach. For instance, all reading 
items taught in a given program are presented in the same format so that 
they can be taught with the same procedure. 

Although PTR is programmed tutoring, the teaching strategy em- 
ployed should be quite different from conventional progratnned tutoring. 
Rather than seeking errorless or nearly errorless learning by providing 
initial cues followed by a fading of cues, the PTR tutor should practice 
minimal cuing at first, followed by increased prompting until the tutce 
deternines the correct response. For instance, on each lesson the stu- 
dent should be presented all of the items within a lesson. After the 
firsc attempt (known as a run) at all items in the lesson, the sequence 
of successive steps is determined by the child's pattern of success or 
failure on iLcms in the first run. If errors occur, additional runs 
she lid be presented, giving only items missed on preceding runs, unLiI 
the SLudenL completes all items satisfactorily. Then all of the items 
in the lesson slrould he presented again and the process repeated until 
the student completes all oi the items correctly in succession, or until 
ten runs have been attempted. The tutor then goes on to the next lesson 
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The order in v/hich the lessons are presented is dictated by 
the Master List. The general pattern of lesson presentation is cyclical 
Several consecutive reading lessons are followed by a few comprehension 
or word analysis lessons. The lessons increase progressively in vocabu- 
lary, in length, and iA task complexity, but each lesson builds upon 
skill.3 acquired in pro'/ious ones. 

An important component of the instructional process is fre- 
quent and immediate feedback. Each tutoring kit instructs the tutor to 
•'Reinforce and go to Step " following a correct response. Only posi- 
tive reinforcement, which includes occasional use of the student's name, 
is used. Only the lack of positive reinforcement should be indicacive 
of an incorrect response. 

In summary, there are eight general aspects of programmed tu- 
toring, as presented in the Tutor* s Guide, Ginn 360 (one of the six tu- 
torial kits available). Annotated descriptions of the general aspects 
are given below. 

• Programmed tutoring requires active learning --The 
student in PTR learns by actively reading and re- 
acting to what he reads, not by passively listening 
or being told. He reads and follows printed instruc- 
tions, and he reads and chooses among alternative an- 
swers. Through the entire tutoring session, he 
should be actively learning as he practices the vari- 
ous reading skills. 

• Programmed tutoring is individualized teachin^ --The 
rules that the tutor follows are the same for every 
student, but these rules allow the tutor to treat 
each student differently, depending on his individual 
ability as reflected in his moment-to-moment suc- 
cesses or failures in reading and in comprehension. 
As a result, progranmed tutoring allows each student 
to progress at his own rate. 

• Frogranmed tutoring requires learning by discoverv -- 
Each program is designed to help the student discover 
the answers to reading- problems or questions by him- 
self, rather than having the tutor tell him tht an- 
swers. Each program begins with a test that presents 
the student with a reading problem or task in its 
most difficult form. If he cannot solve the problem 
in this form, it should be progressively simplified 

by changing its form or by providing more information, 
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hints, or additional context. However, these changes 
and prompts should never provide a complete solution 
to the problem, so that in doing, what a test step re- 
quires there is always some element of discovery. 

• Programmed tutoring emphasizes success --Throughout 
all o£ the progrcuns, a student's successes should be 
emphasized by praise and encouragement. His failures 
should be ignored in the sense that the tutor does 
not call attention to them (other than recording them 
on the record sheet). The student is simply taken to 
the next procedure in the program, usually one de- 
signed to teach the correct responses. 

• Programmed tutoring provides the child with clear 
evidence of proRress --Each student's progress is tied 
to his own successes, which should be clearly em- 
phasized for him. Each student competes only with 
himself so that those who progress slowly should not 
be discouraged by comparison with others. 

• Programmed tutoring is systematic teaching ^-The 
overall objectives are to teach sight-reao . ug, com- 
prehension, and word analysis. Each of these objec- 
tives is systematically represented in a revolving 
sequence of lessons. Each lesson builds upon previ- 
ous ones, and. each lesson must be mastered before 
going on to the next. 

• Programmed tutoring is efficient teaching --Teach ing 
time is concentrated where it is needed. Each stu- 
dent should progress quickly through material that is 
difficult so that a minimum of time is wasted in 
"teaching" what he already knows. 

• Programmed tutoring is human teaching --Ideal ly^ the 
good teacher is patient, sensitive to the student's 
need for success, tolerant of failure, and pains- 
taking in matching her teaching procedures to the re- 
quirements of the inciv^idual student. In PTR, these 
virtues have been progiammed. 



A . 3 . i . 6 R-3 Instructional l rt>gram 

The goal of Project R-3 is to improve junior high school stu- 
dents' reading and math skills and, by providing them with success expe- 
riences, to improve students' self-image. This will help them to succeed 

142 



in school and in the world of work. Toward this goal, the program em- 
ploys individualized instruction in math, reading, and, social studies, 
in combination with a laboratory approach using learning centers. What 
distinguishes R-3 from other individualized instruction programs is its 
emphasis on motivational field trips and gaming/simulation activities de 
signed to demonstrate the applicability of classroom learning to prob- 
lems faced in the* world outside of school. 

For three 45-minute periods t zh day, all students at one ju- 
nior high school grade level receive iu :ruction in math, reading, and 
social studies classes in which each student is on an individual prog- 
ress program. Diagnostic tests are admiriisLered in the three subjects 
throughout the school year to determine each student's areas of strength 
and weakness. Based on the diagnostic tests, a program of instruction 
that will meet individual needs is prescribed for each student. 

Learning contracts are the substance of the individualized in- 
struction program. Contracts that emphasize specific skills and con- 
cepts are negotiated with each student. In conference with the teacher, 
the student agrees to complete a certain amount of work over a desig- 
nated period of time (usually one week). Although contracts are used in 
all three subject areas, they differ somewhat in format and use from one 
subject to another. 

Math contracts for each content area (e.g., multiplication, 
fractions, percents) consist of several subdivisions of instruction and 
exercises in graduating degrees of difficulty. When the student has com 
pleted a section of the contract, he is tested on the work covered. If 
there are skills he has not yet mastered, they should be reviewed and ad 
ditional exercises assigned. When he is able to complete the final test 
on a contract successfully, he moves to the next step of his individual 
program. A variety of reinforcing activities should be used in conjunc- 
tion with the contracts. For example, programmed instruction kits are 
available tor supplementary work, and each classroom should have a sup- 
.>ly of games that require the use of the particular skills the student 
is acquiring. 

Reading contracts are developed around a central theme (e.g., 
science fiction) and encompass activities for all ability levels, from 
which assignments appropriate for each student should be made. Each ac- 
tivity carries a specific number of points, and the number of points 
earned dotcmiaes the student's grade. When a student successfully com- 
pletes his individual assignment, he can choose to do additional activi- 
ties on the contract to increase tlie number of points' he earns and 
thereby improve his grade, a reading contract might include several 
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activities on different ability levels under each of the following cate- 
gories: exercises in punctuation, spelling, vocabulary, dictionary 
skills; readings from one or several literature series with questions to 
answer; original compositions; games that require the use of specific 
skills': and crossword puzzles. The materials that students need to com- 
plete the various parts of the contract should be set out around the 
room at learning centers where students can work together or individu- 
ally, as the activity dictates. 

Social studies contracts are similar in format to the reading 
contracts, and students carry out assignments for points (and grades) at 
learning centers. A contract might include tasks such as report writing 
or suggested topics, readings with corresponding questions, games such 
as matching people to events, an exercise related to the vocabulary in 
the reading, and perhaps a math activity related to the unit being 
s tud ied . 

Games and simulations are used principally in social studies 
classes^ although they are also used in reading and math as motivating 
and reinforcing activities. They should also be used to illustrate the 
relevance of material learned to situations that are encountered outside 
of school. Since simulations are designed to integrate the three sub- 
ject areas and to reinforce skills the students have learned, they make 
use of matli and readinjj, skills with a social studies theme. For example, 
the gaming/simulation activity called "Hurricane Warning Game" is used 
during a social studies unit on weather. It iust-uctn students to make 
decisions about whether to secure their towns je,aLn?i;t Hurricane Eva, 
which is hovering offshore. Each student seJects a specific town for 
which he is responsible. Knowing the cost of securing the town and the 
dollar amount of possible damages as well as the probability that the 
hurricane will strike his town, the student must make his decision based 
on tlie conputation of probable savings of securing versus not securing 
the town. We then throws dice, the sum of which will determine whether 
the liurricane struck his town or not. 

SimulaLicr.is that incorporate difficult math co.iLepts or vocabu- 
lary sliould bo introduced in math and reading clas.ses. .'here instruction 
c..u\ h^'. j^irj.'.n to prepare students for the activity. 

Like !.l.e ^,.injini;/simulation activities, field trips are an im- 
portant motivational force in tlie R-3 program- Trips should be arranged 
luOM lime '.'J cLuit- a: funds permit to point up l-ssons learned in tlie 
^ia..^.rou:n. 1 ur .^:.a;.i^> : e, dfter map-reading and route-planning lesson:,, 
stade'.ts i.;cr/ :.-e ^.tkca .a a bus trip they luive planned thcT"...uIves to 
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Extended field trips of two or threp. days, called intensive in- 
volvements, should be planned for fall or spring to a suitable site 
where students and project staff can spend one or two nights away from 
home and school. Intensive, involvement is a series of learning experi- 
ences built around a particular theme and includes gaming/simulation ac- 
tivities. Preparatory activities should be conducted in advance of the 
study trip, and follow-up activities should build on the exper? "^nces of 
the students at the intensive? involvement site. Such trips become the 
highlight of the year for both teachers and students, many of whom have 
not been away from home previously. The informal atmosphere of the out- 
of-classroom experience encourages closer relationships between teacher 
and student than can develop in the more formal classroom. 

All classes should offer a variety of instructional approaches. 
Lessons are given individually, or in small groups^ and sometimes to the 
entire class. While most of the contract assignments are carried out in- 
dividually, some tasks require students to work together in small groups. 
To avoid the possibility of students becoming bored with working indepen- 
dently, teachers should periodically devote a week or so to activities 
^hat involve the classroom group as a whole. Because the individualized 
instructional program requires teachers and aides to spend so much time 
with individuals, students should sometimes be divided into small groups 
with student leaders who answer the questions of their group members; 
this approach allows the teacher and the aide to devote their time to 
students who have more serious problems. 

A. 3. 3 Cone lusion 

The cor.p]eted program descriptions, which reflected the modifica- 
tions resulting from our own internalization of the PIPs, as well as the 
revisions and clarifications agreed upon at the Washington conference, 
gav^ us an overview of what trie tryout projects should be like during 
the second-year site visits; however, areas of instruction for some PIPs 
were still unclear. The PIPs lacked detail about s':me of the proce- 
duices, especially how uhe materials were to be used and assigned and how 
an individualized curriculum would unfold in a well-impl<^mented classroom. 

A. 4 Site Visit 1 

4.A.1 Data Col lect.^on 

As stated in SectXL u 1^ for our first site visit of the second 
year, we selected a sample of teachers to interview and classrooms to 
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observe so that wo could pursue implementation of the instructional pro- 
grams in depth, rather than repeat the first year's evaluation of the 
general level of implementation. Our evaluation plan was based on the 
assumption that the major program characteristics observed in the first 
year of implementation at each project would remain much the same in the 
second year and that little new information would be gained from a 
second-year analysis in which data were accumulated on every teacher at 
every project. It seemed likely that more information would be revealed 
about the effects of instruction on students' success or failure if we 
looked more closely at only a few classrooms or treatment groups, partic- 
ularly those that fell at the extremes of implementation. Therefore, we 
chose those treatment groups that had indications, on the basis of the 
first year's observations, of being either rather well or rather poorly 
implemented. Since the first year's observations had focused on how 
closely teachers followed the PIP specifications for setting up and man- 
aging the classroom, how they interacted with students, and the clarity 
of their presentations of lessons, our implementation judgments were 
made after a consideration of those areas of the program. Table 4-2 
shows the number of instructional staff observea and interviewed. 

Many features of the instructional program were not well described 
in the PIPs, but were nevertheless essential for a site to reproduce in 
order to call its progran implemented. The specifications in the PIPs 
were concerned mainly with space, furniture, instructional equipment and 
materials, and the adult-student ratio. Our observation procedures had 
to be sensitive to the classroom processes that were likely to influence 
achievement test scores. Otherwise we would have no documentation to 
justify the expectation that the MAT results would be relevant to PIP im- 
pacts . 

The program descriptions presented in Section 4.3.2 were used, uo 
guide the development of PIP-specific observation instruments for the 
first site visit. Although the instruments were PIP-specific, each fol- 
lowed the same general plan, which allowed us to document the instruc- 
tional treatment experienced by students and the implementation of PIP 
speciEications for both classroom management and curriculum resources. 
After pretesting and revision, the final observation instruments included 



In R-3 suites where many of the teachers were new to the project in its 
seconu ^ ear of implementation, sui*. a judgment could n^t be made. 
Thererore, the sample for these sites was chosen on the basis of 
fitting some teachers from each subject area into a workable interview 
a- 1 observation schedule for the site visitor. 
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Table 4-2 
OBSERVATION AND INTERVI^ S.\MPLE 



PIP 


Project 


Number of Teachers 
in Sample 


Catc.b-Up 


Bloomington 


4 




Brookport 


2 




O CI 1 d A. 


9 




Providence Forge 


2 




Wayne City 


2 


Conques t 


Benton Har.bor 


3 




C 1 eve land 


3 




G loversvi He 


4 


HIT 


Lexington 


3 




Olean 


4 


IRIT 


Bloomington 


3 




Ok 1 f'lh oma C i t" v 


■X 




S chenectady 


3 


PTR 


Canton 


6 




Dallas 


6 


R-3 


Charlotte 


7 




Lake Village 


3 




Lorain 


5 




Schenectady 


5 


Total samp le 




70 



means for documenting the following indicators of implementation of in- 
structional techniques: 

• Classroom features and resources. 

• Classroom management before and during the instructional 
period • 

• Student grouping arrangements. 

• Individualization of materials and subject matter. 

• Summary of teaching techniques of the instructional staff. 
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• Summary of student behaviors and responses in the instruc- 
tional setting. 



Most of the sections of the observation instrument were designed 
for straightforward recording of occurrences; however, some sections nec- 
essarily required more judgment by the observers. 

One of the areas in the PIPs that was generally left unspecified, 
or was at best unclear, was that of how the curriculum materials should 
be used in an ind ividual:' /-f:d instructional program. The sectioii of our 
observation instruments called ''Individualization of Materials and Sub- 
ject Matter" required the observer to record the materials that selected 
students were working on during the class period. Before we observed in 
classrooms, we asked the teachers to consult their records for four or 
five students and to tell us what materials and lessons they would be as- 
signed during the class period in which we would be observing. With 
this information, we could verify whether students were working on dif- 
ferent materials and levels (as prescribed by an individualized progr^un) 
and whether the selected students were in fact using the materials as- 
signed to them. We could also determine whether students were using the 
materials appropriately, that is, whether they were actually engaged 
with and working on the mat --rials. From these observations, we could 
judge whether or not learning was taking place in the classroom. 

In our interviews with teachers, which followed the observation pe- 
riod, we identified changes that had occurred since the firsu year rela- 
tive to roles, training, and the use of materials. We also questioned 
them about their interpretation and implementation of PIP concepts such 
as "individualization," "core materials," and "diagnostic-prescriptive 
procedures." We asked them to describe a typical instructional period, 
including how they made assignments, what materials they used and why, 
and details of their record-keeping systems. 

In our view, we could not make a plausible argument that the PIP 
projects would influence MAT scores unless we investigated the curricu- 
lum that the teachers wure using. For this reason we completed, on the 
first site isit, a "F)ictionary of Core Materials" with each teacher we 
interviewe"' and observed.'' The dictionary listed and described the mate- 
rials spec-" Lied in the PIP, as well as materials that had been observed 



"in R-3 sites, the dictionary was completed on math materials only be- 
cause the PIP specified i more manageable number Oj math materials than 
read in;, materials. 
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in classrooms during the first year of the field test. The site visitor 
and the teacher located materials in the classroom, and the site visitor 
checked them off in the dictionary* Teachers were asked to designate ma- 
terials they considered to be "core" materials which, for this purpose, 
we defined as materials used with 50% of the students or used 50?o of the 
time* In addition, we asked teachers how they divided the materials 
into teaching segments (i.e., chapters, sections of^chapters, particular 
pages) and recorded that information in the dictionary. The completion 
of a "Dictionary of Core Materials" for each classroom enabled work to 
begin on a curriculum analysis that would tell us whether the PIP proj- 
ects used materials whose content was covered in the MAT battery. 



4.4.2 Review of Data from Site Visit 1 

Afcer the site visits, we attempted to use the data we had col- 
lected to classify projects relative to their performance on the insLruc"* 
tional component of the PlPs. We had, however, made two errors that prc- 
vented^us from satisfactorily doing this. The first error was that we 
had thought it possible to assess implementation globally, at the proj- 
ect level. The PIPs had been designed to create projects, and this 
seemed the appropriate level of abstraction at which to evaluate imple- 
mentation. Unfortunately, at. that level of abstraction, the degree of 
implementation was not directly observable. The site visitors were un- 
able to verify directly that the project was implemented; they could ver- 
ify only that some of the project staff were behaving as specified in 
some respects, but not in others. 

Since what was observable were project staff interactions, the ten- 
dency was to refer to how well individuals were doing'w^iSv:At this level of 
detail, site visitor reports were objective, in that they were based on 
direct observations. At the project level, their reports were clearly 
subjective, in that they had to "sum up" their judgments about the staff 
to reach a conclusion. 

Our second error was that of r.ot putting the descriptions of the in- 
structional component of the PIP on our observation instruments in the 
order in which instruction usually occurred. We found it difficult to de- 
termine the success of a teacher on activities that we did not antici- 
pate would occur. It was also difficult to distinguish events that were 
not specified in the PIP from ambiguities, the teacher ^s resolution ot 
which interested us. 

The next section describes the steps wc took to overcome these weak- 
nesses in our original procedures, 
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4.5 Identification of Gaps and Anibir?:uLt:Les in the PIPr 



Since a major objective of this year's field-cest evaluation was to 
find those aspects of each PIP program that contributed to improvements 
in student test scores^ we felt a need^ based on the interviews and ob- 
servations of the first site visit^ to reorganize our descriptions of 
the instructional components for each PIP program. The organization 
used on the original site visits made it difficult to verify directly 
that the specified techniques were being used. Reorganization would 
give the details of the instructional procedures in the order ^ which 
they werc^ used in the classroom. Such an exercise would forcu k \:o 
think through areas that were still vague and would perhaps givtu v 
better grasp of the programs for our final visit to the project: si' ^ 

We also hoped that with rhe new organization we could more edoiiy 
discriminate variations in the learn ing/ teaching process across si.::.;-o 
within PIPs. These vari.itions were expected to be explanatory fac!: ::s 
for the effects of the i IP prDjects on student achievement, as well as 
suggestive of revisions for the PIPs, which were being rewritten for dis- 
semination. 

As an outline for reorganizing the descriptions, we used a typical 
teaching plan" that broke down the instructional functions into the fol- 
lowing steps: 

• Diagnosis of each student's needs 

• Prescription of materials designed to meet diagnOo-r^f! nt.eds. 

• Presentation of the lesson/skill 

• Guided practice; independent study 

• Assessment of progress (monitoring. c.'Sting) 

• Reinforcement activities 

• Motivating techniques . 

As we organized the descriptions into these steps, the programs 
took on the essential structure and order that had been missing previ- 
ously, yet there were still areas about which we were unsure. What did 
become quite clear, however, was where the PIPs were vague or allowed 



A teaching plan used by the Cleveland Conquest instructional staff was 
adapted for this purpose. 
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the teachers a great deal of freedom, and what areas of instruction the 
PIPs omitted entirely. Now we could designate those aspects of the in- 
structional programs that were ambiguous, free to vary, or not specified 
in each PIP. For each step of the teaching plan, we developed a matrix 
for each PIP on which we listed the revised descriptions. Discrepancies 
between our descriptions and the PIP guidelines were identified and ex- 
plained under the column headings, "not specified," "ambiguous," or 
"free to vary," as appropriate. The matrix developed for Project Con- 
quest is sliowri in Table 4-3, which illustrates the method used to clas- 
sify descrepanc iss for each PIP. 

The completed matrices' served several purposes. They clarified f ;»r 
us the reasons why we, and the tryout project staffs, had felt secure 
with the raatiagement aspects of the projects, but had been baffled by uhe 
instructional programs. We had attributed our confusion to the organiza- 
tion of the PIPs, since directions were scattered throughout the compo- 
nents in scmetimcG unexpected places and in differing amounts of Jetail. 
New we were ablo to shov; graphically the gaps and ambiguities in the 
PIPs that had resulted in our less - than-complete understanding of the 
programs. The matrices easily allowed us to identify weak areas in the 
PIPs and uo generate suggestions for revisions that would help clarify 
the packaged programs for future users. In addition, they provided the 
basis for our final visit's interview questions and guided the develop- 
V''\nt of our observation Instruments for the second site visits. 

Examples of gaps and ambiguities for each PIP are described below 
to ':^iv6 the reader a better understanding of the areas that we felt were 
of special interest for assessing the degree of implementation. 

The Conquest PIP specified several activities that had to occur in 
t're labs so tb.at each student could participate in three or four activi- 
ties, r^aiLy. As shown in Table 4-3, the PIP did not present the details 
of c i. Dsroom management that would help teachers plan the class period 
to incorporate the specified activities, as well as to give individual 
attention ::o each child, handle problems that might arise, and keep the 
detailed records that the program demanded. Since the implementation 
of Conquest's individualized instruction program depended on efficient 
cl v^sroom management, one of the speical foci of our second site visit 
interviews and observations was on teachers* orgainzational and planning 
skills . 

The Calch-Uo PIP recommended the use of a wide variety of materials 
along with the Random House Criterion-Referenced System for diagnosing 
needs and prescribing related materials. The use of the criterion- 
referenced materials, however, was not explained in the PIP, and the 
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ERIC 



Vroftritn AnptH t 



Uuidtfd praccico 



Progress :iHs»4sm(>nC 



■ . torcemenc 



Mottvrttln^? technLfiues 



i.tiided practue with 
tt.'aclier; student is to 
rt'ccivc sume imlividu.il 
at U'nt ion . 

Use ot c,irr«'ls lor intlo- 
p^-ndiMit work, tor •ii>.ne 
activitiiis iJttfii fitutlcnts 
iiTti confuii'fit eiiouv.lt. 



I'se iif ti'achinj^ machin«'S 
and dif V i L I'S ; d.i i I y use 
advised. 

Student schedules and 
classroom manaKtrnent : 
Huw louK students should 
spend on a nub) eel area 
e«ch day. Readinj? room 
— sradi'S 1-3; phonics, 
10 min ; basal trxt , 10 
nin; prov'rarmtied readintj, 
15 min; inral ri'adinp, 
j!.imi!9, t«>achin>: machines, 
10 rain. Clinics — prades 
^-b: proyrflmmcd rendinp; 
10 min J comprehetiHion, 
10 min; vocabulary, *> 
nin; .si>:ht words, 5 min; 
!»'.T;hinK machines, b 
min; oral readinv;, iO 
min . 

Koles of clinic ian : 
tionityr, tutor, diag- 
nostician, mot I vator , 
ohbi.-rver, prescnbi-r, 
nrjfafiizfr. 

R»*cord kfeping: fold- 
ers, notebooks, daily 
record sheets. 



Symptoms . 



In. struct ions 
as necessary. 



tod taKno.se 



Hf»lpase of the .student 
from treatment: post- 
test; consu I tar. ion b»?- 
twrjen project director, 
<: 1 J n ic i an , and sope r- 
v i sor . 

(iame dav i uae of f>am»fs 
that refjuir** s^udent» t< 
niie skills trifd durLn^i 
the w»»ck or al sume pri" 
vious t irre . 



Cam*: dav: 1 day per 
week. Thursday after- 
noon and Fridav morning 
su^lKesred , 

Providinn nucceas expe- 
ri«»nces: .Makinji assii?n- 
mentM that students can 
do, 

P'-a»sin« stitdent« tor 
••verv littt" thiuK. 

L'fl»f o f .ich i evem<»nt 
aw.trds : for itrad L.a t • "n , 
extra reading, honoi ablo 
mention. atten(*ance. 



Omit tird/Not .Sp,-c i ; i 



How ti» i-ronp srudents 
while teacbinv;. 



How t4» (Icttfminf* when 
students ari- r<*a,fv iiow 
tt> j»si;.i s e t j fi' t i ve ver - 
su.s me t ft:*; 1 1 ve iiMe ot 
carre 1 a . 



How to manage the cla.sa- 
room within the ^uidi'- 
1 i nt>s , 

Amoun t ot I I>-x i b i 1 i t v 
a I l«we f .imonv, clinicians 
and amunii student.s . 

How to .schedule stmlents 
around rchool schedules. 



How to perform in the 
var louii rolf;s , 



How Rood records help 
t cac her K ma iti c a i n awnre- 
ncss ot student progress 
Descriptions of ouch 
records. 

How records relate to 
objectives. Discussion 
of var ious techn i ques 
and importance of the 
r»^cord keeping. 

Mean i nft , Symptoms o f 
succes s or prob 1 cms . 



When to rcdiapnose stu- 
dents , How to d id(;nose 
after initial intensive 
dia^noiits. How to use 
results ut initial di- 
aKnosift thrrughout the 
year , 

How to uii? posttest re- 
sults. How and when 
s til den t stiou 1 d he re- 
1 e jsed f rum projjram. 



f(*iires for rr 



iiiforc 



Hiju ctmi'-'an pr'jvules 
i nterm i 1 1 prt. rev i ew of 
skills previously taught 

.Methods of motiv,itin,T 
stutlent'i who do not re- 
spond , (Little informa- 
c ion prnvidod on how i ti 
motivate students ,it 
all. ) 



Anih itf ti,)u> / Ct)nt r .id i c tor v 



Not clearly described. 



Kre« 



V.irv 



Wnetlier teacher wt>rr... 
with groups or wi' i 
d i V i J iia 1 s — left t . . 
judtfment , 

How much time a parti 
11 1 a r i> t u d e n t wu r k s in 
r,he carre 1 --teacher ' b 
dec i s i oil , 



Wha t mjch 1 nfS to u»e 
with each sCudent. 



Decisions regard inj; lev- 
els within an activity 
for each student --c I in- 
ician's decision. 



Kecords de«criptions 
va^ue , Not c book and 
folder contents ambig- 
u }\iH . Some commerc iai 
examples were in PIP. 



Vague item called symp- 
tomatology found in 
Project Director's Di- 
rectory. 



Objcure , 
pafl s i nK , 



Mention»»d in 



K>iitii{ m an <»bsciire 
place in rh.' PIP. 



Additional means of mot i 
vatitiR HtudentH wneii 
necessary-- teacher ' s 
iec i 6 ion. 



I'se <if arhi 
us needed-- 
dec i s ion . 
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Random House system was keyed to verv few o£ the PIP-recomn,endc>d materi- 
als. Presumablv to help the projects with this problem, the PIP in- 
cluded infomation that had been reproduced from a set of index cards 
used at the originating; site to identify thoso sections of certain mate- 
rials that were relevant Co specific skill deficiencies. Unfortunately, 
the index cards were also keyed to many materials that were not recom- 
mended in the PIP. Therefore, an analysis of each of the PlP's core ma- 
terials would have to be conducted by project staff to determine how 
they related to diagnosed needs- For teachers inexner ienced both with 
the criterion-referenced system and with the PIP-recommended materials, 
this would be a fairly complicated and very time-consuminp task. 
Yet to implement the Catch-Up progmiu as specified, the task would be es- 
sential. How the criterion-referenced, diagnostic-prescriptivo proce- 
dures were handled was of special interest in the Catch-Up second site 
vi -.it, since it was an area that would be considered in a judgment of 
teachers and of well and poorly implemented projects. 

Thp HIT PIP had few ambieuities or omissions. Tlie Instructional 
program was specified in enoueh detail to enable project staff to meet re- 
quirements for tutor training and classroom management, and the Pif- 
recommended materials incorporated adeauate explanations of how they were 
to be used. The principal omission was the lack of explicit specifica- 
tions for the instructional pace that tutor-tutee pairs should maintain- 
an essential ingredient of the originating site's program. Because the 
pace at which lessons are conducted contributes to the degree of interest 
xnd enthusiasm with which students attack their work, the HIT observa- 
tion instrument included an assessment of instructional pace in each cen- 
ter. The activities of the adults during the tutoring sessions were 
also recorded on the observation instrument, since the PIP specified 
that they should be circulating as monitors throughout the session, 
helping as needed, but not interrupting the tutor-tutee interactions. 

• The IRIT PIP recommended .two basic diagnostic tests for the first 
and last weeks of the cycle and specified that teachers select and admin- 
ister other appropriate diagnostic tests during the cycle. The PIP did 
not explain how to select tests appropriate to the subject matter .and 
.tndenc levels or how often to administer the tests. Neither did it ex- 
plain how ins', r.ct-ional materials were to be used to teach the diagnosed 
skill deficiencies. Therefore, as focal points, the IRIT interviews in- 
corporated items relating to teachers' choice and use of diagnostic tests, 
how thev prescribed materials and activities for individual students, and 
how they maintained records of needs and progress for individual stf.dents, 

Like HIT, the PIP for PTR was strai gl. - forward and, when used in com- 
bination with' the specified self-explaining materials, gave enough 



154 



details of instruction to enable the staff to set up and maintain the 
program with ease. For this reason, the PTR interviews and observations 
were focused on determining to what degree the specifications were fol- 
lowed, rather than on the resolutions of ambiguous statements or omissions. 

The R-3 PIP specified that the individualized instruction program be 
structured around learning contracts and that they be individually nego- 
tiated with students. Individual negotiation encourages students to as- 
sume responsibility for their own work and gives them an understanding 
of the importance of setting realistic goals. Yet the PIP gave^ con- 
flicting information about con*:ract format and use: The PIP stated that 
R-3 Teachers were supposed to develop their own multilevel, multiactivity 
contracts, but, in fact, a series of math contracts, each devoted to a 
single area of the math program (e.g., fractions), which were developed 
at the originating site, were specified as part of the core materials. 
The PIP stated that each contract should incorporate a posttest, but the 
sample reading contract reproduced in the PIP had no posttest; the PIP 
stated that contracts should contain predetermined grading jschedu les of- 
fering the student a choice of grade to be achieved, but the core math 
contracts did not contain grading schedules. Interviews with the tryout 
project teachers included questions on the use or development of con- 
tracts in each subject area, and the observations recorded whether and 
how contracts were used in the classrooms. 

Interview guides and observation instruments were revised for each 
PIP before the second site visit to focus more sharply on those factors 
that would differentiate teachers, not projects, on how well they imple- 
mented the PIP instructional component. When the site visits were com- 
pleted, we used the data that were encoded in the field to assess the 
teachers' degree of implementation in terms of the PIP specifications, 
as clarified by the Washington conference. However, we did more than 
collect data that would allow us to judge implementation solely on clear 
PIP specifications; we also collected data that would allow us to judge 
whether projects resolved ambiguities in conformity with the project phi- 
'losophy, as gleaned from the package and the conference- 

4.6 Site Visit 2--Data Collection 

The revised observation instruments were developed to record the 
functions of the teaching plan discussed in Section 4.5. We observed 
and recorded how the instructional process was handled in the classroom, 
including those areas that were ambiguous or not covt:red in the PIPs. 
We subsequently interviewed the sample teachers to learn how they han- 
dled these ambiguities. As on the first visit, each instrument was 
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PIP-specific, but all instruments were ne.- organized into the following, 
general categories: 



• Classroom management and grouping arrangements 

• Individualization of materials 

• Classroom facilities arid atmosphere 

• Classroom features and resources 

• Student behaviors during instructional period 

• Teaching techniques of the instructional staff. 

The observation form allowed us to register whether the PIP-specif ied 
structure was there and to document teaching practices that we judged 
"good implementation" or "poor implementation.'' 

Examples of our collection procedures are shown in Exhibits 4-1 and 
4-2, which are reproductions of records made in an R-3 clasGr.>om on sec- 
ond site visits. The students were working independently and in small 
groups at learning centers on contract activities that required reading, 
writing, research, and some math skills. Information was recorded about 
student interactions and working behaviors and about their activities 
and materials (Exhibit 4-1) as well as about how the teacher spent his 
time (Exhibit 4-2) during the observation period. The observation rec- 
ord shows that instruction in this classroom was being implemented as 
specified in our instructional program descriptions . 

How teachers interpreted those parts of the instructional program 
that were ambiguously stated or missing in the PIP was the emphasis of 
the interviews, but we also questioned them thoroughly about how they 
managed each step of the program. We asked teachers generally how they 
planned an individualized program for each student and specifically how 
they determined, for example: 

• On what skilli to put each student to work. 

• V/hat materials each child should use. 

• What instructional approach is best for each student. 

• Whether to introduce new material to individual students 
or to small or large groups^ 

• Whether a student is learning or not and, if not, what to 
do about it. 

What motivates each student. 
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Exhibit 4-1 



STUDENT BEHAVIORS DURING INSTRUCTIONAL PERIOD 
Time 

(Most Students) 



1. Students feel free to approach 
adult for help 

2. Students ask each other for help* 

3. Students wait for adults to offer help' 

4. Students initiate task-related questions 
^. Students initiate nontask-re laced questi 
D. Students quiet and orderly 

7. Students converse but do not 'disrupt 'class 

8. Students disrupt entire class . 

9. Students appear interested ir. task 

10. Students appear enthusiastic about 

11. Students appear bored with t.isk 

12. Students appear rrstless 



task 



Most 


Some 


None 


(D 


2 


3 


1 




3 

CD 

3 


1 


2 
2 


I 


2 






2 


3 




2 


3 


1 


2 

2 


6) 

3 


1 


CD 




1 


2 




1 


2 





As 


Not as 


Not 


Specified 


Specified 


Specified 




2 
2 


3 


1 


2 
2 


3 


1 

CD 


2 
2 


3 




2 


3 




2 


3 




2 


3 




2 


3 


2 


3 




2 


3 



13. Average number, of activities per student 
during instructional period 



N/A 



14. Approximate minuces of each activity 

15. Materials anJ nctivities: 



Students work nn individual assignments 
Class works -,n a variety of materials 
5^-Cudents work .n materials in small ^rouos 
students work on m.-Uerials in large ^voup< 



16. 



Activities Occurring; 



a. Students art- working on contracLs 

b. Students .ire vorking on games 

c. Students ;irv working on simulation ! ^ 



Aii 



1 " . (.'ominen c s : 



4+ 


5 


(20+) 




4 


5 


Yes 


No 




2 


CL 


2 




2 


1 




Some None 


2 


J 


2 




2 


Q 


Ye ."^ 


No 




2 
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TKACUEK INSTRUCTIONS. TECliNiqi'K 



(Circle appropriate number) 



1. Teacher stays with students, holping 
as needed 

2. Teacher circulates, helping 

students as needed 

Teacher remains in one place and 
students '^o lo adult 

4. Teacher works with individual 
students 

5. Teacher works with small groups . . . 
6*. Teacher work witl) entire class . . . 

7. Teacher spe;uls fi e wiLi^ otlier 
adults (<?i>0>f-Mr't^>J 

8. Teacher spends tine at classroom 
management 

9. Humor is evident between students 
and teacher 

10. Aide spends time at classroom 
management 

11. Aide helps individual students 

with assignmer.t ^ 

12. Teacher corrects students 
unacceptable behavior 

13. Teacher gives positive feedback 

to students for their work • • • • ' 

14. Teacher gives inappropriate feedback 
to students l\»r their work 

15. I'eacher siyle 

16. Teacher is a facllicator 

17. Teacher is controlling 

18. Teacher lacks control of class . . 

19. Teacher participates in activities 

20. Teacher di-rr^ards activity and/or 
c"ni Idrei^ ' s t».,-ed^ 



21, 



1 cache I s p 

wi th t he S LIM'.-Ir 

Teachtr''. plrinin/ is sonewh<>t ev 
with slMcietU:-. 



(( [ic\i'. appropriate number > 

Lhc. period is cU.arly evld,-.U hci Interactions 



idem through bur i nt ev-.ict ions 

I-.achor's l.ck M- .lannin^is evident through her iMt .rac C ions ^ 
with st.idents , 



with 

Ch,mw;uv^ ot AcL ivi ties: 
Te.acher displays ilexibLlity in reasstgnntM 
individual sLudenl* 



it of work to 



Yes 
. 1 



No 



Most £ 


1 omc ^ 


one 


As 

specified £ 


Not as 
)pecified i 


Not 
Jpecif ie 


■ 1 ^ ( 


> — > 


3 


_ . 1 — . 


2 


- \ — 


CP 


2 


3 


G. 




'X 

J 


1 


2 








3 


1 
1 

1 


2 

a.- 
*> 


3 

■J 


CI ' 
1 


•} 

2 
-) 

2 


3 

3 
3 


1 


2 


d ' 


1 


2 




1 




3 


1 


2 




1 


CD 


3 


J.' 


2 


3 


CO 

1 


2 


3 
3 


d' 

1 


2 

2 


3 


1 


cT> 


3 


cv 


2 


3 


1 


2 






2 


3 


CT' 

1 
1 
1 


2 
1 
2 
2 


3 

1 


1 

u 


2 
2 
2 
2 


3 
3 


1 


2 


(J) 




2 





N/A 
/ 3 ' 
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Interviews and observation data were used to generate a classifica- 
tion scheme for describing those teacher implementations on which analy- 
ses of curriculum and instructional materials would be conducted^ as de- 
scribed in nhe following section. 

4. 7 Assessment of Implementation and TeacVier Responsiveness 

Following the second site visit, wa completed debriefing tasks in 
preparation for a final selection of treatment groups whose curricula 
and test scores './ould be analyzed in detail. The selection process re- 
quired a judgment of whether the students whos'^. classes we observed and 
whose teacher we interviewed were learning in the way the PIP intended^ 
and whether the t;::cher's implementation of the instruction. program 
was in accordance with the PIP's axplicit specifications aid with the 
programme philosophy of teaching and learning. These judgments were 
formed and justified by the information collected in our interviews and 
documented on our observation instruments. In making our judgments of 
implementation^ we used that observation data for reference as we wrote 
a description of the instructional program of each teacher, emphasizing 
how the teachpr handled the gaps and ambiguities in the PIP specifica- 
tions imd describing her/his implementation of each of the steps of the 
teaching plan. Then, from the descriptions of individual teacher imple- 
mentation^ we constructed a generalized description of the instructional 
prograni at each project^ citing examples to support judgments and noting 
exceptions. Variations in implementation r • tss projeccs within PIPs 
were designated and probable causes explaii. . including management^ 
training^ contextual resolutions;, professional experience of the teach- 
ers^ and the like. These summaries provided the basis for our recommen- 
dations for PIP revisions. 

With the completed descriptions of individual teacher implementa- 
tion and the generalized description of implementation at each project^ 
we rated each teacher on the bnsis cf overall performance on two dimen- 
sions: 

• Implementation^ as shown by: 

- Attention to the explicit specification in the PIP (evi- 
denced by the degree of fidelity to the specified proce- 
dures and activities) . 

- Understanding of the program philosophy (evidenced by 
the method of resolving the PIP*s gaps.^nd ambiguities). 

• Responsiveness^ as shown by: 
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- Awareness of (inch student's progress during the period. 

- Awareness of each student's interests (evidenced by 
teacher having time for personal comment). 

- Amount of time teacher was working with students (versus 
timt^ spent on classroom management tasks). 

- Amount of time students waited for individual attention, 
(Some sat with hand up for 5 minutes or more; others 
stood in line at teacher's desk for 10 minutes or more.) 

- Quality of attention. (Some to.achers listened carefully 
to students' explanations of problem; some jumped in and 
explained things the student already knew and left be- 
fore the real problem was solved.) 

Along both of these dimensions, teachers were rated "good/' "so-so^" or 
''poor." Below we present some specific examples, using the same areas 
of investigation referred to in Section 4.5 to state our rating proce- 
dure. To maintain the confidentiality of our data, we will speak in gen- 
eralities. However, in fact, each teacher rating was considered in de- 
tail on a teacher-by-tcacher basis. 



4.7.1 Assessment of Implementation 

How Catch-Up teachers handled the criterion-referenced system of di- 
agnosis and prescription contributed to their final ratings. Some of 
the experienced teachers seemed to grasp the concept immediately and 
were able to systematize their procedures so they could easily designate 
the specific parts of a variety of materials that were relevant to diag- 
nosed skill deficiencies. These teachers were rated "good." On the 
otlier hand, some of the less-experienced teachers never fully utilized 
the criterion-referenced system. They used their own informal assess- 
ment of students' strengths and weaknesses or depended on a few materi- 
als with which they felt comfortable to meet every student's needs- 
They were not able to coordinatu the diagnostic and prescriptive steps 
in the lesson plan, so we gave them a "poor" rating. 

The Conquest PIP failed to describe how teachers could organize 
their classes to incorporate the several activities and record-keeping 
responsibilities that were supposed to occur daily. Because Conquest's 
individualized program could not function well without careful planning, 
accurate and up-to-date records that relate objectives to tasks, and effi 
cient management of a variety of simultaneous activities, all teachers 
were judged on their organizational skills. Good classroom management 
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was necessary but not sufficient for judging a teacher "good/' and other 
features of her implementation had to be considered in the final rating. 
However, poor classroom management necessarily implies a "poor'* rating 
for implementation because a poorly organized Conquest program could 
not be well implemented. 

The instructional pace of HIT tutoring sessions was implied but not 
described in the PIP. Since the rhythm of tutor-tutee interactions was 
a good indicator of whether learnin^^ was taking place, the implementation 
of HIT teachers was judged, among other things, on the ability to keep 
the tutoring sessions moving along at a pace that would encourage the ac- 
tive participation and enthusiasm that characterize High Intensity Tu- 
toring. In a center where tutors knew their job and were conscientious 
and quick about cuing, reinforcing, catching errors, and tallying re- 
sponses, and where the teacher circulated and helped but did not engage 
in lengthy explanations (which break the rhythm of the tutoring session), 
the teacher was given a "'good" rating for his understanding of the pro- 
gram's intent. In a center where tutors and tutees seemed apathetic or 
engaged in off-task conversations^ and where the teacher was attending 
to record-keeping tasks rather , than circulating around the center, the 
teacher was rated "poor" on his program implementation. 

Although the IRIT PIP did not give directions for using the diag- 
nostic tests or the variety of materials it recommended, the diagnostic- 
prescriptive procedures and an efficient record-keeping system were essen- 
tial to the IRIT individualized instruction program. IRIT teachers were 
interviewed in depth about their use of these procedures and were judged 
on their implementation of them. A teacher who was judged '*poor" on 
this aspect of instruction did not use any diagnostic tests because he 
"felt" that his students needed help in all skills^ and his prescrip- 
tions consisted of rotating the students through a set of selected mate- 
rials. A teacher who was judged "good" on this IRIT feature used highly 
systematized diagnos tic -prescript ive procedurtis that enabled him to spec- 
ify what pages of a variety of instructional materials were related to 
skill deficiencies diagnosed in testing. His students were advised of 
the skills on which they needed to work. 

Because the PTR instructional program v;as clearly specified by the 
PIP and the materials it used, it was possible to judge implementation 
without the confounding of ambiguities. Tutors were judged on how 
closely they followed the directions in the programmed materials during 
the observation period. Although the materials contained a carefully de- 
lineated test so that everything the tutor should do or say is explicit, 
including feedback, some tutors wore judged "poor" because they deviated 
from the text^ used informal teaching methods, and employed negative 
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feedback. Their students Vs'ere given confusing instructions and appeared 
to be intimidated by the tutors. Tutors judged to be ''good/' on the 
other hand, adhered to the programmed instructional methods, pmvided 
positive reinforcement, and demonstrated an interest in their students' 
vvfork. Tliese tutors seemed to ]iave captured the spirit and intent of PTR, 

The development and use of learning contracts in the R-3 program 
was one of the ambiguous areas on which R-3 teachers were judged. Some 
of the. reading and social studies teachers, despite tight schedules that 
afforded little i)repara v Ion time, designed contracts that included activ- 
ities for a variety of ciChievement levels and a predetermined grading 
schedule, while other teachers neither designed nor used contracts. A 
few teachers attempted to negotiate contracts v;ith students^ but the ne- 
gotiations were quickly dropped because students did not set realistic 
goals and the experience proved disappointing. The result was that 
those teachers who used contracts made some assignments of specific ac- 
tivities and then allowed students r_o do further work if they had time 
and wished to earn a higher grade. In effect, the students were 
learning to set their own goals and to take responsibility for their own 
work, so although tlie contraccs were not negotiated, the solution was 
compatible with the intent of the program. Therefore, those teachers 
v^^ho designed and used contracts in this way were judged "good," while 
those teachers who used no contracts were judged "poor" on this instruc- 
tional feature. 

4.7.2 As se s smen t of Re s pon s ive ne s s 

Tlie responsiveness of the teacher was judged on her interactions 
with her students. For example, one teacher was judged unresponsive 
when one of lier students sat idle for 15 minutes before being noticed. 
Another teacher asked a student to read aloud, then walked away before 
the student had finished; this teacher was judged unresponsive. In sor.? 
classes, up to one-thirtl of the children were observed to i)e waiting lor 
nev; assignments . 

One teacher judged "good" on responsiveness ?^<^.emcd to "have eyes in 
the back of her head." She gave three groups reading tests, but v.'vis 
awcire of what, each student was doing, and assigned additional work as 
necessary. In another class, a group of students was observed sepa- 
rately with two teachers. The site visitor reported that one would not 
know Che two groups were the same. One teacher monitored the studeat 
so that tiiev were i.iucli more task persi.^tent and less restless. This 
teaelior was judgc-d to have good responsiveness. Th'..- other teacher was 
rated "so-so. 
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When a final judgment had been made of each teacher we observed, a 
numerical rating was assigned to each teacher according to the scheme 
shown in Table 4-4". We interpret this as a nominal classif icatioii 
scheme . 



Table 4-4 

CLASSIFICATION SCHEME FOR PROJECT TEACHERS 
DURING OBSERVATION 



Teacher* s Implementation 
o£ Proiect 


Teacher ' s 
Respons iveness 


Good 


So-So 


Bad 


We 11 imp lemented 


1 


2 


3 


So-so implemented 


4 


5 


6 


Poorly implemented 


7 


8 


9 



4 . 8 Conclus ions 

Table 4-5 shows the number of teachers given each rating, by proj- 
ect. Despite the lack of detail in the PIP specifications for the in- 
structional program, at least 807o of the ratings were of an acceptable 
degree of implementation. It would appear that project directors did se- 
lect the best teachers they could find for the project and that most in- 
structional staff did grasp the PIP philosophy and intent. Only 9 of 
the 71 responsiveness ratings (13%) were "bad," and only 13 of the 71 
ratings (187o) fell into the "poorLy implemented" category. 

It is not safe to generalize from these ratings because we did not 
sample teachers. We hoped to get the best and the worst of thein. It is 
clear, however, that the sites and PIPs did differ on their implementa- 
tion. In Catch-Up we were unable to find any unresponsive teachers. 
Gloversville Conquest had no we 1 1 - implemented project teachers. No 
teacher was found to he implementing R- 3 well. 

Our interpretation of these variations is dealt with more fully in 
Volume One. Here, we need only remark that the type of package rcpre- 
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'[•KACllER RATINGS, BY- PKOJL.CT 
(ill Nuiuhers of Teachers) 





Rat in^s" 


Pro iect 


I 


') 


3 


4 


5 


6 


7 


8 


9 


Catch-Dp 

B loomi.nv; toil 
Brookpor t 
Galax 

P r o V i cU Ml c F o r ^; e 










4 





























1 


1 
















1 


















Wayne City 










. I 






- 1 




C onques t 

Benton Harbor 
C leve Ivirul 
G Loversvi 1 le 


1 


1 




I 












■ ) 








I 


















I 




1 


1 


2 


HIT 

Ll ington 
e^lean 


I 








I 








I 


1 


I 




— 






1 


1 




IRIT 

B 1 oorninc, ton 
OkLah.oma City 
Sclicnec taci y 


■ } 






I 
































1 






I 








I 


Cvin ton 
Del I las 








2 


2 




X 




2 




1 






I 










Char 1 t.U tc- 
hake Vi 1 hi^/.e 
Lorviin 
Sci!enei:tv!dv 








6 


J 






















I 


2 








2 


3 


















4 


I 








Tola 1 


14 


5 


0 


14 


22 


3 


3 


4 


() 



"Orif LvMciu/r is r.iL^'cl Iv.ice: once lor ro.ul i ii;; .iml oiu-e for ni,iLli.'i;i,il Ics. 
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seoCed hy all tlu* PlPs, oxcopL R-3, could be successful under the condi- 
tions of our tryouLs. The packages were successful not -r'-rely in Llia j 
teaci^ers mechanically followed directions, but also in tliat Lhe sense 
and philosophy of the oroject was present at the s i. . .- 

We do noL conclude ihaL the package was itself sufficienL to in- 
tro'Hice superior projects. These sites were fairly intensively moni- 
tored. Further, the fact that USOE paid for a conference in Washington 
do:'bcless both motivated the participants and gave thorn information that 
was not in thi- P1P> and not Ln the official modifications, which are 
attached in A[^pendLx (;." 

Insofar as these particular projects are concerned, in this section 
we have answered two o i' the issues suj-.gested by the cu rr icu lum- re f erenced 
analyses. We have defined the PLPs' instructional programs, and wc have 
a.'. sensed the dciu'ee to which eacii of our observed teachers was Imp lemen t Ln g 
tha t rv-'L'/.'arti . 

The noa section discusses the actual materials used In the PIP 
classe-' and our prc^coduros in ascertaining these materials, and ht.)w we 
assessed the connect botween the materials and the >\AT . 



It should be noted that in addition to the fact that the first year 
findings seirvod as the basis l:or revising tho PlPs, our seccMid year 
f i n d i n g, s w e r e s I ] a f e c w i 1 1 1 Kl-lC as the study p r (.) g r c s s e d . 'i 1 1 e s e f i n d i, n j • ;■; 
have been usod as i:^.jv,it to the new packages currently being implemented. 
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3 ANALYSIS OF CURRICULUM AND TLST CONTLNT 



5 . 1 Introduction 

The first set of liuld opo rat i ons - ~ to determine the degree of impLe 
mentation of the instructional prac t i ces - -was described in the previous 
section. In tliis section, we describe the second set of field opera- 
tions and associated amlyses. which were designed to determine whether 
the PIP-speci f Ted currlr'ilum materials were being used in the field-test 
project}:; and whether t;)-- tests being used to measure ^ s tuden ts ' perfor- 
mance corresponded with the curriculum to wliicli the students were ex- 
posed. 

If tiic skills tested by the M/\T v/ere not covered in the PIP- 
spccific curriculum materials used in th.e projects, we would have no rea 
<:on to attribute MAT achievement gains to PIP projects. Cursory exan.ina 
tion of tluj M\T in the first year suggested that, in several cases at 
least. >[AT and PIPs were not well matched. Ideally, a detailed investi- 
gation of this issue would require: 

• Obtaining a record of the lessons that cacli student cov- 
ered in each PIP-speci f ied curriculum material. 

• Determining the skills required for passing each item on 
the >L\T, for each level administered to students in botii 
pre- and post-tests. 

• Deten^uning l\\c skills tauglit In each lesson in the PIP- 
speci fie l1 ( a n d us e d) c u r r i c u I urn ma t e r 1 a I s . 

The test of PIP project e f fee tivenuss v;ould then require locating stu- 
dents who failed items in the fall and who subsequently covered PIP- 
specified curriculum material tliat was relevant to those items. All 
other things being equal, we 1 1 - Imp lementcd classes would be expected t:) 
show a greater proportion i)[ students answering tiie failed pretest items 
correctly on the posttesc. If this expectation was not fulfilled, we 
would have no evidence to claim that PIPs "worked" because a positive as- 
sociation betv;een degree of implementation and outcome would be absf.nt. 

however, we v;e r e n t sure i f i n f o r ina r i o n c ( n 1 1 d be ( > h t a i no d o n i 1 1 d i - 
vidual students. Moreover, we did not know if an analysis of tlie curric- 
ulum materials would be feasible, or how much c^f a test v;ould be IcTt i )' 
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we restricted the MVl" to only tliose items that were relevant to the cur- 
riculam materials covered. Consequently^ we attempted analyses at two 
levels: an intermediate- level , where we found PIP-specified curriculum 
materials that wore relevant to types of skills included on MAT subtests 
and that were used wicii students; a detailed level, where we matched a 
student with the Items he had covered. 



5.2 Data Collection Loiustics 

We attempted first to assess the feasibility of obtaining each stu- 
dent's record of instruction on a daily basis. These data could be ag- 
gregated to determine general use of curriculum materials; if the rec- 
ords were detailed enough, they could be used for the individual 
student- level analysis . 

The feasLbrlity of obtaining individual student records was investi- 
gated in the fall of the 1975-76 school year. In November the evalua- 
tion project analyst visited four tryout projects* to determine whetlier 
individual records were being maintained on every student in the project 
(as most of the PIPs themselves required) and, if so, how complete and 
usable the records were. Most .eachers did have available written les- 
son plans or scliedules of instruction (SOIs) with individual assignments 
Cor each student. 

We already knew that PTR and R-3 projects would not conform to this 
pattern. PTR projects maintained cumulative records with no association 
bv^tween lessons and dates, and several R-3 projects submitted records 
without dates. In R-3 we decided to concentrate on the mathematics 
classes because the curriculum could be more systematically tracked. 
The R-3 mathematics contracts were approximately the same as assignments, 
and each student had a record of contracts completed. 

During thL' regular site visits in December, we showed a model SOI 
to the Pip'instructioiia}. staff, detennined that teachers or tutors 
wore maintaining plans or records containing similar information, and 
then requested that tiiese schedules be saved for us to examine during 
our evaluation. During the December visits, each site visitor developed 
the Dictionary of Core Materials. As described earlier, this dictionary 
listed materials that the instructional staff claimed to use most fre- 
quently and that therefore served as core curriculum. Our guideline 
ior such macerials was tlia t they had to be used at least 50% of the 
time or witli at least 50;' of the -tudents. Kach entry in the dicticniary 
was identifiiMj i)y exact title, pub Li.sher * s name, date of publication, 
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series, set, level of material used, and- other information nece^'sary 
for us to obtain exact duplicate items or sets of materials. 

In January, our site assistants collected a week's accumulation of 
SOIs from project directors or directly from instructional staff. The 
site assistant then sent either machine copies or the originals (if 
teachers did not wish to retain them) directly to SRI, Since abbrevia- 
tions were used extensively in the notations on the SOIs^ each teacher 
was asked to provide a key to abbreviations used. The" key enabled us to 
match the entries on the schedules with the more complete entries in the 
dictionaries of core materials. 

After examining the SOIs received from all PIP project instruc- 
tional staff for the week^ we saw that the schedules were likely to per- 
mit a determination of materials actually covered by the students." In 
January we notified teachers not in the observation sample that ve would 
not be collecting any more SOIs^ but we request"id sample teachers to con 
tinue saving their schedules for collection at posttest time. In early 
March^ we asked the sample teachers to give their January through Marcii 
schedules to our site assistants for transmittal to SRI. 

Finally, at the time of spring testing^ we asked the sample instruc 
tional staff to submit their remaining SOIs for the period through the 
Friday before test administration. Thus^ while we did not collect sched 
ule5^ for the entire period between pre- and post-test^ we hoped to have 
a good idea of the r.iaterials used by students from early January to the 
April test date. 

Obtaining SOIs presented some difficulties. Ahout h^lf of the 
sites had to be telephoned and reminded to turn in some portion of che 
data. Most were simply slow in sending in all of the schedules for the 
period fi-om January to posttcsting. At the two sites that turned in 
schedules for only tho. latter part of the period^ site assistants were 
asked to search teachers' records and submit the missing schedules. By 
mid-June we had received the SOIs from all sites. 



Although SOIs for the lab programs were generally written as plans^ not 
as records of what students actually covered^ many teachers noted on 
these sheets when assignments were not ccmiplcued. These plans or sched 
ules were proxies, not ideal records of materials covered. They wcre^ 
however^ available and imposed only a slight data collection burden on 
instructional st-if!.. 
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5. 3 Completeness of the Data 



before using the SOIs in any analyses, we attempted to determine 
the completeness and quality of the data collected. Table 5-1 shows the 
number of individual student schedules received for each week. The 
right-hand column in the table shows the -average number of schedule? re- 
ceived per week over th.e number of students with valid spring tests who 
were in the sampled teachers' clashes. Ideally, the numbers should be 
approximately equal; that is, the number of schedules submitted per week 
should not vary much from the mean. 

Tabl.^ 5-1 shows that the number of SOIs received for a week was of- 
ten greattjr than the number of students* in our sample because teachers 
were asked to s'end schedules for all students in their groups • Not all 
students, however, were included in our study. For example, we received 
schedules for between 45 and 47 students in Providence Forge each week, 
but there were only 34 students in our evaluation sample. The remaining 
11 students were dropped because they exited from the program before 
posttesting, or had an invalid posttest. 

We had expected data for IRIT, for one Catch-Up site, and for PTR 
to be submitted differently .from data sent by the other projects. Be- 
cause we had included in the tested evaluation group only those IRIT stu- 
dents in the second, or middle, cycle of the program, we were interested 
in the schedules for only that group. Thus, the period for which we ex- 
pected schedules in IRIT projects was not January to April, but the ap- 
proximate ten-week period corresponding uo the middle cycle at each site. 
An additional peculiarity was that each IRIT student had three teachers-- 
one for each of three content areas; thus, expected three schedules 
per day per child. 

The Catch-Up staff in Galax were making systematic, cumulative rec- 
ords of what each student covered Ln each curriculum material during tlie 
year. Wo accepted these records rather than ccllectLng, aggregating, 
and interpreting the daily schedules ourselves. 

Finally, as mentioned earlier, in PTR each student's performance v;as 
recorded by the tutor on a Record Sheet, Thus, instead of a daily rec- 
ord of ins ruction, each student had a complete, cumulative record of ma- 
terial cov^cred. Both PTR sites submitted these records for every stu- 
dent in t:he pro'oct, not simply for those in tiie sampled tutors^ group. 
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Tabic 5-1 gives evidence of incomplete dacu^, as discussed belov.^: 

• Of the four teachers sampled in B locmlrigton Catch-Up^ one 
teacher wa.*-. i;ot asked to keep or retain SOIs for submis- 
sion to SRI and on-? turned in records for only three weeks. 

• One teacher in Benton harbor Conquest did not begin to sub- 
mit records until February 2-6. Another did not begin un- 
til February 9-13. The tiiird teaclier had a few SOIs for 
January, but did not keep good records until February. 

• Two of the teachers in Bloomington IRIT did- not submit 
SOIs for^December 8-12 nor fcr February 23-27. 

• The teachers in Schenectady IRIT were not asked to retain 
SOIs until the sixth week of the second instructional cy- 
cle. Only two of these teachers turned in schedules for 
the remaining four v/eeks of the .cycle. 

• Charlotte and Schenectady R-3 project teachers did not in- 
clude dates on thei r SOIs . 

• The only K-3 mathematics teacher in Lake Village did not 
turn in many schedules prior to the second week of Feb- 
ruary . 

• One teacher in Lorain R-3 did not submit any schedules;, 
and the other teaclier did not include dates on the sched- 
ules. 

Thu Gloversville and Olean SOIs allowed some major findings about 
tiiu implementation of the respective Conquest and HIT projects in those 
sites. The schedules received from Gloversville contained no weekly in- 
formation because they were not kept on a daily or weekly basis. This 
was our first indication that students there did not attend the lab on a 
regular, f ive-days-per-v/eek basis^, but rather on an irregular basis as 
assigned by their classroom teachers. Entries on the schedules were 
spotty or incomplete; presumably the regular classroom teacher rather 
than the PIP clinician maintained records of student progress. In any 
case, the luimber of schedules received v/as larger than the number of stu 
dents in our sample; the reason is that the students in two of the teach 
err>' classes were excluded from our evaluation sample because they had 
T'l^eeivt'd so little of the Conquest treatment that it was considered un- 
f/fii to uvaluatLi the IMP on the basis of their performance. 

In Olean, v/e received records for 233 students thou};ht to he in \ \\c. 
project on liio specified four-days per v/eek, Ilov/over, 130 of thcst! .stu- 
ilents wt:rL- sehedulo.l in the lli.T classes only every other week. 'i'haL iS; 
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one or two teachers alternated student groups each week. We had not de- 
tected this scheduling pattern in our earlier interviews and observa- 
tions. For the 185 students with valid test data, an average of 165 rec- 
ords were received. 

Although Table 5-i reveals a number of irregularities^ we received 
from most teachers in most sites a fairly complete and reliable record 
of the materials covered. Only one entry gave us reason to question the 
validity of the data: Although March 15-19 was Easter vacation for Lex- 
ington;, one teacher there -submitted schedules for 25 students. Perhaps • 
she had made plans for students for that week befc.ce realizing that it 
was a vacation period, but the quality of. the entries on these schedules 
and our knowledge of the teacher's organizational style led us to be- 
lieve otherwise. Exhibit 5-1 shows four schedules of instruction re- 
ceived from the field. They vary in completeness and format^ but three, 
of them (with their keys) contain u&able information. Schedule A (on 
Exhibit 5-1) does not contain adequate information about curriculum mate- 
rials. The teacher who prepared this schedule also received low ratings 
on implementation and responslvein's s during our interview and observation 
visits. This example reflects a general observation we made about the 
data. SOIs that were incomplete or of poor quality were submitted by 
the same instructional staff in our sample who also received the lowest 
ratings on implementation of the PIP instructional program. Thus, qual- 
ity of curriculum information on the students is confounded with degree 
of PIP implementation. Any possible bias in the remainder of the analy- 
ses would lie with the well-implemented instructional groups or the 
'*good" teachers because more of theiV data were usable in our analyses. 

In general, we believe that enough SOIs were received for the en- 
tire sample group for the entire period and that they provide fairly re- 
liable information about what was being covered in the PIP projects. Al- 
though wo received information for about 1700 students*, clearly we did not 
have a random sample of student lesson assignments. Nevertheless, SOIs 
were a fairly direct measure of what curriculum was used in PIP classes 
and were clearly superior lo interviews and observations alone. 

5 . 4 Congruence rjcLweu?n JUP-Spe c if icd Curriculum Materials 
and MateTials Used m tne Field -Test Pr ojects 

For each project, we compiled a list of all equipment and curric- 
ulum materials that appeared aim^rhere on llie SOIs. The number of titles 
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EXHIBIT 5-2 (Concluded) 



C. D. 




ranged from 2 L*or Dallas PTR to 65 for Schenectady IRIT. With each list 
we addressed the question: Did the project staff use the PIP-specif ied 
materials? Tlie problem (as reported in Section 4) was that the materi- 
als specifications in the PIPs were often ambiguous. We decided to list 
all the materials mentioned in each PIP and designate them as "specified" 
materials, with a distinction made between "core" materials and "supple- 
mentary" materials.''' These were the "PIP-specif ied materials" against 
which we compared the list of materials generated by aggregating across 
all SOIs collected from the sampled teachers. 

Our findings from this comparison v/ere that: 

• Some overlap existed between what was PIP-spec if ied and 
.what was used; that is_, some of the. PIP-speci f ied materi- 
als were used . 

• Many of the materials specified in the PIP were not used. 

•. Teachers used a surprisingly large number of materials 
that had not been listed in the PIPs. 

Tables 5-2 th.rough 5-6 summarize our findings about the instruc- 
tional materials used at each project site. In each table^ the left- 
hand column indicates the number of PIP-specif ied materials used and the 
percentage, of the specified instructional curriculum this represented 
for each project. The right-hand column indicates the number of non- 
PlP-specif ied materials used and the percentage of the total instruc- 
tional materials this represented for each project. In addition^ a sepa- 
rate entry shows the number of materials^ either specified or 
nonspecified, used by all the sites With the same PIP program. This num- 
ber is also included in each project's figure<^. These data shovj that 
not every site using the same PIP used the same combination of specified 
materials, nor did sites reject the same combination of specified 



We could not always determine which materials were "specif ied«" Wlien 
tv/o materials v/ere called by the same name, we counted both as speci- 
fied materials. Tor example, McGraw-Hill Programmed Reading and Sulli- 
van Reading Prograr^ were both referred to as "Sullivan" by teachers. 
(McGraw-Hill Publishing Company had bought out BRL, publishers of tlie 
Sullivan Reading Program, and had issued a completely revised series.) 
Another di f f i cu Ll.y" was with the Catch-Up PIP, whicli referred to the SKA 
Math Kit. There v;as nothing by that title, r. Itliough there was an SRA 
Math Learning; Svstoni and an Diagn,osis: An Im^ I, rue L iona 1. Aid- 

M.i Un'MKi I i c s Villi c'h cmild li.ivc fit i\\c iiia t: c r i 1 (icsc r i bed , and l-ht- re- 1 iM'f 
LM I 1 ii • !' ' ii''-' 'P'-''^ i 1 i ^'d ;^iat rr\.:\ 1 . 
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T'lblo 5-2 



SPECIFIED AND UNSPECIFIED MATERIALS 
USED AT CATCH -UP SITES 



by Projeci 


S]H'C i t i ed 


Not Specified 
nnd Us ed 


Used 


Not Used 




7, of 
Total 
Spec i f i od 


N 


I of 
Totn I 
Spec i f ied 


N 


7o of All 
Materials Used 
at Eacli Site* 


Co i\' in. J t 0 r i n 1 s 
Bl oom i ny\ t otT 
Brookpo rt 
C.I l ix 

P rov i (li-n W^ vi'.r 

Wa viu' Ci t y 

Us Oil by .ill s i t.*.' ^ 


b ion. QV. 

2 )3. 3 

3 50.0 
3 50.0 
■4 b6 . 7 
1 1 b . :^ 


0 0 *; 

4 66.7 
3 50.0 
3 50.0 
2 32.3 


33 75.0% 
23 79.3 
1 7 7 7.3 
12 63.2 
lb 72.7 
2"^. 2.0 


To L :\ 1 no . CO r n .'vt r i 1 1 s 
s pt'c i f i ed = b * 


To t - J 1 no . 

core u'^t' i 
- b- 




To tn I no . un sp ec i - 
fied materials = 
84 


Supp 1 omen t .i rv l . v i .i ! s 
Bl oom In'tX t on 
Brookport 
Cal-iv 

P ro V i (U'n vt' Fof^:.' 
W.i yno C i t. v 
Us.^d hv ill i t 


5 35. 77 
2 J), (■ 

2 1 -'f . 3 
2 8 . 

2 : -4 . ^ 


I M 71.4 
12 85. 7 
10 71. 4 
12 85.7 




To».j1 no. siipo I L.! rv 
m:j Iv V i .} Is p^'c i f i .'d ^- 


loL.il no. 
supp. nsfd 







TliG ratio between t lu^ number of "not specified and used" materials and the 
total number of materials used at a site. (Tiie total includes all materials 
used at a site, both spec i f ied- -cor*.* and supp lemcntary- -and unspecified.") 

Cr.'di* for ..nii' iiiUcri il e uh i r.ivt'n I '^ ! p.-icher-madp materials, ii/imes, 
iivl fr-.'.- r-iiiai.; • iho'iyji j.nnv tin Ld.Mi r i f i .-d tna r r.' r i 1 1 fuav have })eMi 

M • I'd '-'I 1 !M n 1 . !! I i : I-,-'.: 



l!-. ! 'i.it'S on- f v:-.- 



l-.q'i t. p:'.i'.'!r. 
men t ! on r<\ in 



' J 1 I - - 



i'-v! in rtinimu'ion with m-o' .if m 1 1 . a- i a 1 - 



1 17 



2 i . 



Table 5-2 (Coucludea) 
b, M.'iti. 



Ml r h Mn I er i .1 1 s , 
bv r ro i et- 1 


Spec i f ied 


Not Specified 
and Used 


Used 


Noi Used 


V 


% of 
Totnl 
Spec i f led 


N 


I of 
Tol.ll 
Specified 


N 


% of All 
N.iterinls U.^ed 
nt Each Site 


Core m.i L cr i .i 1 s 

W 1 oom i n L', Ion 

I* r-.i V i dence ForL'.c 
Us '.*d i>v ,1 1 1 silos 


\ 100. 0 

1 33.3 

2 66. ' 

3 100,0 
I 3 ^ 


0 n % 

0 0 

2 66.7 

1 3 3.3 

0 0 


6 54.57, 
7 7.8 

10 83.3 

7 70.0 
15 78.9 

V'' ■ 2.1 


Tot ; no . .. iM'i' iTM t t' r L 1 1 s 
s ^- i f i (• d = 3 


Tol.il no. 
coro ust'tl 
■-^ 3 




Toi.'jl no. 'an.speci- 
f i ed mci I or i n 1 s = 3 


Siipn UiiU'ii t. ii v in.uori ils 
B 1 .^oin inj:t on 
B rookp(^ r I 

P rov id en cr Ftu'tie 

W.i %'n»' (M t V 

Used bv ill >; ii 0^ 


2 '-4 0.07. 
1 20.r; 
1 20.0 
1 20. f) 
1 20.0 
1 20.0 


3 60.0"/. 
A 80.0 
A 80.0 
80.0 
80.0 




To t .) 1 no . snpp 1 I'lii-jn t .n rv 

r.M I r i .1 1 s sp.'c- i i* i - 
»- 

■) 


T^^it.il no. 
'^tipp . us od 
- > 


1 





I for .'.r. o U-r i .1 1 ■■.icb is {'.iv<.n to teacher-made materials, j:;aines, 

.ml '"i"' r.ilinv' '^ven t boin::i inanv an i cb.'P. t i f i ed mntorials may have been 
u s f'i.i v i • b i ii 'M rb . a t i>>;o rv . 

Tbt' ha I'fb'-a I . / ^< O t '-M r. \vul-a4V' did not clearly indicite vh<'tber orn* of 

the nat'M ials referred to vas the Si^A Mat'ii Learning System or the SKA 

Ma. tb^'ir.a t i er. I) i .ii:.n(.v-; i s : tbereforo, each was considered a specified 
:iia t e r ill. 
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SPECTFTKD AND UNSPECIFIl'D MATERIALS 
USF.D AT CONQUEST SITES 



Reading Matori.'ils 
by P ro i ec c 


Specified 


Not Specified 
and Used 


Used 


Not Used 


N 


7o of 
Total 
Specified 


N 


% of 
Total 
Specified 


N 


% of All 
Materials Used 
at Each Site 


Core mnterials 
Benton Hnrbor 
CI e V 0 1 n n d 
Glovorsville 
Used by all sites 


16 61.3% 
16 61.5 
8 30.8 
5 19.2 


10 38.5% 
10 38.5 
18 69.2 


36 66.7% 
40 69.0 
55 84.6 
3^' 2.1 


Total no. cow materials 
specif led = 26^ 


Total no. 
core used 
- 20 




Total no. u . i s p e c i - 
f i e d materials - 
119'' 


S u pp 1 emen t a r y ina t g r i a I s 
Benton Harbor 
CI eve laud 
Gloversvil 1 e 
Used by all sites 


2 33. 3Z 
2 33. 3 
2 33.3 
1 16.7 


4 66.7% 
4 66.7 
4 66.7 




Total no. suppl omen tary 
materials specified = 6 


Total no. 
stipp. u.sed 
= 3 







Credit for one material each is given to tencher-made materials, j^ames, and 
free reading; even thouf:;h many unidentified materials may have been used 
within each ratep^orv. 

t 

Includes six types of equipmcMU. Th.ree materials mentioned in the hardware/ 
s o f { wa r e p a c- k a e a re co n I a 1 n e d in the W e b s t c r C' I n s s r o om R e .i d in g (. 1 1 n i c hu t 
< 1 re CO u n ted a s s e p a r .i t e m a I" e r 1 a 1 s . 
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Tiblo 5-4 



spf,cifi?:d and unspecified materials 
used at hit sites 





Specif ied 


Not Specified 
and Used 




Used 


Not Used 


Core Materials 
by Proiect 


N 


% of 
Total 
Spec if ied 


N 


% of 
Total 
Spec if ied 


N 


% of All 
Materials Used 
at Each Site 


Core matorl.3ls--rGading 
Lexington 
Clean 

Used by all sites 


3 60.0% 
3 60.0 
1 20.0 


2 40 . 0% 
2 40.0 


6''' 66.6% 
3 50.0 
3 7.7 


Total no. core materials 

4- 

specified - 5 


Total no. 
core i:sed 
- 5 




Total no. unspeci- 
fied materials = 
8* 


Core ma terials --ma th 
Lexingtw 
Clean 

Used by al 1 s Ltes 


3 25.0% 
3 25.0 

2 ' 0 


9 75.0% 
9 75.0 


3 50.0% 
3 50.0 


Total no. core materials 
spei: i f i-cd = 11 


Total no. 
core used 
- 4 




Total no. unspeci- 
fied materials = 
6 



^1. 

Credit for one material each is given to teacher-made materials, games, 

and free reading even though many unidentified materials may have been 
used within each category.. 

^Cne material (Conquests in Reading) was included in the hardware/ 
software package by mistake but was used in Lexington because they 
had problems obtaining the Sullivan Programmed Reading material. 
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Table 5-5 



SPECIFIED AND UNSPECIFIED MATERIALS 
USED AT IRIT SITES- 



Reading Materials, 
by Pro iec t 


Spec if ied 


Not ^Specified 
and Used 


Used 


Not Used 


N 


% of 
Tot.'^l 
Spec if i-jd 


N 


% of 
Total 
Specified 


N 


% of All 
Materials Used 
at Each Site 


Core materials 
Blooinington 

Schen ec tady 

Used by all sites 


10 18.2% 
7 12.7 
5 9. 1 
2 3.6 


45 81.8% 
48 87.3 
50 90.9 


34 63,0% 
2^ 63.2 
50 76.9 
3* 2.5 


Total no, core materials 
speci f icd = 55 ^ 


Total no. 
core used 
= 12^ 




Total no. unspeci- 
fied materials = 
92* 


Supplementary materials 
Bloomington 
Oklahoma City 
Schen ec tady 
Used by all sites 


10 12.2% 
7 8.5 

10 12.2 
5§ 6. 1 


72 87,8% 
75 91.5 
72 87.8 




Total no. supplementary 
materials specified - 
82*"^ 


Total no. 
supp. used 
= 17± 







Credit for one material each is given to teacher-made materials, games, 
and free reading even though many unidentified materials may have been 
used within each category. 

t 

Includes one type of equipment, 

± 

Equipment use was not always specified in conjunction with use of 
materials mentioned in the schedules of Instruction, 

All are eqtilpmont. 

Includes 13 types of equ ipmon t . 
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Tnble 5-6 



sPECiFiKi) an:) unspecified materials 

USED AT R-3 SITES 





Specif ied 


Not Specified 
and Used" 




Used 


Not Used 


Math M.ucrinls , " 
b V V ro J 0 c t 


N 


7o of 
Total 
Spec i f ied 


N 


% of 
Total 
Specified 


N 


% of All 
Materials Used 
at Each Site 


Core rr.Mtori.ils 
Chn riot t e 
Lnke Villai;" 
Lorn in 
Schen cc Lndv 
UsL'd by .ill s i ICS 


2 2b, 01 

2 2 5.0 

3 37.5 
2 25.0 
1 12.5 


6 25.0% 
6 75.0 

5 62.5 

6 75.0 


19 82.6% 
3 42.9 

6 60.0 

7 77.8 


Tot.iL no. core mntorinrs 
s p c c L f i t* t-l = 8 


Total no. 
core used 
= 4 




Total no. unspeci- 
fied materials = 
30 


SijppLom€'n inry PM^orinls 
ChnrloLto 
Uikc V ill apt.' 
j Loriiln 
1 Schenoclady 
1 Used by :ill sites 


2 12.5% 
2 12.5 
1 8.3 

! 0 0 

1 0 0 


U 87.5% 

14 87.5 

15 91.7 

16 100.0 


! -- 


Total no. supplementary 
•natorlals specified = 


Total no. 
supp . us ed 
= 3 







'Ih£. R-3 program also includes reading and social studies ins 
but ve vrre able to adequately assess only the materials use 
matl*. cl isscs. 
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materials. Some projects used more unspecified materials than other 
sites (varying from about 437. to 85% of their materials), and some PIPs 
were associated with more use of nonspecified materials than were 
other PIP?. 



Oata for PTR projects are not tabled^ because they are so ea.sy to 
describe. The PIP required only that PTR projects employ a tutoring man- 
ual that coniplemenLed the basal reading text used in the regular class- 
room. A supplementary material, the Alphabet Skills Book, was also men- 
tioned in the PIP. Both materials were used in Canton and Dallas. (In 
Canton^ however, v/here students enter at the first grade level v/ithout 
kindergarten, no basal readers had been used before the PIP arrived. 
These texts had to be obtained before the PlP-specif ied materials could 
be used as intended.) 

Among other PIPs, Catch-Up ' cts appeared to use the greatest 
p^.!:-..-. at jf specified materials fo. loading instruction, with Bloomington 
ir.::l'^diug 100% of suck materials in their curriculum. Even with the 
overlap of material:, among Catch-Up projects, only one specified reading 
material v/as used ul all Catch-Up sites. All Catch-Up projects used 
many unspecified materials for both reading and math^, with each project 
using a different >et of such materials. 

With the exception of Gloversville, Conquest sites used about 60% 
of the core materials specified :.n the PIPs. A large number of unspeci- 
fied materials v/ere added at every site. 

The 121 reading materials used in IRIT corresponded even less 
closely to the specified lists, and only tv/o of 55 specifiei-i core materi- 
als were used in common across the sites. More unspecified materials 
were used in IRIT than in either of the other two lab programs. 

The Catch-Up, Conquest, and IRIT PIPs all had long lists of materi- 
als. Except perhaps in the Catch-Up PIP, the original Hardware /Software 
Packet did not clearly state which materials were core. .In all three 
PIPs, various materials were suggested and instructional staff were en- 
couraged to assemble a variety of materials and to individualize instruc- 
tion by providing students with materials suited to their needs. The 
Catch-Up PIP stated tlinC each teacher should have his own funds to pur- 
cliase the nuuerials he liked and shouici use materials in any way he 
deemod nracLLcal. 

Like the lab .-;rouranis, che R-3 i'lP encouraged use o 1; a varit;Ly of 
iTiaterials. Evcmi inore than the others, if encouragci'l instructors to seek 
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out materials that were likely to motivate individual students. As ex- 
pected, the R-3 projects did use unspecified materials and did not use a 
large percent of the recommended materia 1:>» 

Except for FIR and to some extent KIT projects, little congruence 
was found between the specified (core and supplementary) materials and 
the materials actually used in the project. This finding is surprising 
for those who expected the instructional programs in sites with the same 
PIP to be the same in terms of materials and equipment used* Insofar as 
comparison of titles permitted us to tell, packages written in the manner 
cf five of the original six PIPs would not promote the use of a common 
set of curriculum materials in new sites. 

we cannot say, however, that projects deviated from PIP instruc- 
tions. Although our comparison of titles shows that "PIPs (except PTR) 
failed to promote the use of exactly those materials and only those mate- 
rials recommended, this was probably not a violation of RMC ' s intent in 
developing the PIPs because Catch-Up, IRIT, R-3, and Conquest PIPs en- 
couraged teacher discretion in choosing materials. 

We are not naive enough to assum*-; that teachers had to use exactly 
the same materials in order to implement effective curriculum.' We know 
that, especially when individualization is required, different combina- 
tions of texts, equipment, and other teaching materials were necessary to 
carry out the intent of the curriculum* Neverthol'iiss, PIPs were pre- 
sumed to present enough information about pedagogical philosophy and 
skill emphases to (1) permit teachers to ascertain the essence of the ef- 
fective curriculum of the original project and (2) promote the use of ma- 
terials incorporating the intended skills lessons in the intended manner. 
With the help of reading curriculum specialists, our next analysis en- 
abled us to assess whether, in spite of the variety of materials used, 
the PIP projects never che less adhered to the curriculum intent of the 
PIPs and covered the same skills in a similar manner. 



Although in this report wc often use "curriculum'' to mean the materials 
and equipment to which students are exposed, we use the term here in 
its general meaning as "planned learning experiences encountered by stu- 
dents," a definition that includes the instructional philosophy ns wcM 
as the knowledge and skills covered in the materials. 

184 
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^ • ^ '^^ ^'talysr.s of the. Core Curriculum at Each Proiect 

Before we could match, project curricula with PIP-specif ied curric- 
uiu, we had to determine what materials were being used as the core of 
the instructional program. We examined the SOIs again for the materials 
used most often. (The list of all materials used by each project was 
too gross for this analysis because once-used titles f>iso appeared on 
this list.) 

After determining which materials were used most frequently by proj 
ect students, we called on reading curriculum specialists to help us de- 
scribe the skills emphasized in each." This gross analysis was con- 
ducted by means of a skills checklist. As stated earlier, our general 
intent was to determine the relevance of the MAT to the PIP curriculum. 
We also intended to use the skills analysis to determine whether, even 
v;hen different sets of materials \;ere used, the project staffs under- 
stood and imp lerrronted the PIP- intended curriculum. 

5.5.1 Pre cedures 

Because we received SOIs for approximately 1700 students, time and 
effort dictated that informati on on the use of materials not be tabu- 
lated from every schedule. Instead, wc» sampled schedules by PIP, proj- 
ect, grade, and teacher/tutor. A sample of five students was picked 
from each grade at each site. 

The grades were t-jrouped roughly according to the MAT battery that 
students received in the spving because we expected to compare the gen- 
eral skills known to be covered in the materials with the general skills 
tested on the Mi\T. The only exception was that we grouped first and sec 
ond grades, even though students in these grades took different tests. 
Groupings were as follows: 

• Grades 1 and 2 

• Grades 3 antl 4 



Consultants were: Ms. Pacricia Bixlcr, Coriricr reading curriculum coor- 
dinator, San Mateo County Schools (California), currently principal of 
Knolls School in San Mateo; and Dr. Arlene Bonnie Tenenbaum, former SRI 
consultant, currently evaluation specialist for Cupertino School Dis- 
trict (California) . 
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• Grades 5 and 6 



• Grades 7, 8, and 9.. 

Besides grade level^ individual teachers were likely to aifect 
which materials were designated as core. At each site, the five stu- 
dents sampled were dravm so as to be distributed across tlie teachers in 
that site and across the grade levels ^xor which each teacher had respon- 
sibility. For example^ in the Bloomington Catch-Up project, our observa 
tion sampl^^ had three teachers. Teacher A had only first graders; teach 
ers B and C had only second graders c The sample included the following 
s tudents : 

• Two first evade students ironi teacher A 

» Two second grade students from teacher B 

• One second grade student from teacher C. 

To tabulate the list of core materials for each project, we used 
the following procedures: One entry was recorded for each material used 
h\' a student each day. If the student received two lessons in a mate- 
riii on a single day, only one entry was recorded for that material. 

The frequency data is shown in Tables E-1 through E-4 in Appendix 
E. The data reflect the full period for which schedules were received, 
that is, from January to posttesting, for all PlPs except IRIT. IRIT ' s 
SOIs cover the students enrolled in the second cycle only. Because 
neither PTR nor R-3 sites submitted SOIs on a daily or weekly basis, 
the procedures could not be carried out for these two programs. 

The ten materials most frequently used by each project during this 
period before posttesting were tabulated. For each PIP, Tables E-1 
throu;-h E-4 show the ten materials used as the core of the curriculum 
by each project. The second column in each table indicates if the mate- 
rial was specified as core or supplementary material in the PIP. 

The individual skills into which the materials could be most com- 
fortably categori/iod were as follows: 

• Recognition of Sounds and Letter Recogni t i.on--Des igned Lu 
teach l.Lter-sound correspondence and visual discrimina- 
tion of letters . 

• Decoding- -Sometimes used synonymously v/i th phonics (associ- 

iting a letter or combination of letters with a sound and 
applying such knov/ledge in identifying words). Lessons 
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are designed to teach word identification and converting 
print into speech. Included are pronunciation and associ- 
ating a group of letters making up a word with the sounds 
in its spoken counterpart. 

• Structural Analysis--Entails looking at words to locate 
parts of them (e.g., syllables^ prefixes^ suffixes^ spe- 
cial endings, root words). Structural analysis may be 
used in conjunction with phonics (phonetic analysis) and 
context clues to identify a word. 

• Vocabuiary--Entails gaining knowledge of the meaning of a 
x<^ord and learning to recognize it in prints (Words in tVie 
curriculum materials were assumed to increase in complex- 
ity and to decrease in frequency of exposure as grade 
level increased.) 

Antonyms and Synonyms--Used to increase vocabulary. 

Comprehension--Entails understanding the meaning of a 
v;rittan word, a written sentence;^ or a written passage of 
one paragraph or more. Responding to questions and acting 
on the information read are included. 

Recognition of sounds and letter recognition. are beginning reading 
skills and are usually covered in the first grade. These skills are in- 
cluied only on the MAT Primer, which is the battery for entering first 
graders. The n^xt two skills, decoding and structural analysis, are 
also beginning reading skills and are usually not included on norm- 
referenced tests for students above the first grade. Vocabulary can 
cover a huge range, depending on the complexity of the words and the fre- 
quency of exposure in appropriate contexts. Vocabulary items are in- 
cluded on every level of the MAT from Primer to Advanced. Reading com- 
prehension also covers a range of skills and interactions of skills that 
have never been satisfactorily understood. Generally tests of reading 
comprehension include passages of incre' ing length and have increasingly 
complex vocabulary and syntax for studc;iLS in first grade and above. 
Reading subtests on the Primary I, Primary II, Elementary, Intermediate, 
and Advanced MAT all test reading comprehension. 

Because we did not tabulc?te information from the SOIs about the lev- 
els in the curriculum series at v;hich the students were performing, we 
asked the reading curriculum specialists for each grade group to check 
only the skills appropriate to the given grade level or below. That is, 
we asked the specialists to ass ume - -especially whe n ana ly z ing cur r icul tini 
series intended for kindergarten through sixth grade levels--that none 
of the studen^ts would be covering materials above the average level of 
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difficulty for their grade* Several other assumptions were also nec- 
essary: 

• That materials were used as intended by publishers or manu- 
facturers Cor full effectiveness (e.g., that both the audi- 
tory and visual components of the Auto-Vance machine were 
used as des igned ) . 

• Tiiat teachers provided the necessary inscruction for each 
studer.c a? outlitied in published manuals and as suggested 
in tha FlPs, 

• That exposure really meant covering the material adequately 
enough to learn tlie skills* 

5,5.2 Analysis 

Tables E-1 through I'-4 in Appendix E show the most frequently used 
materials and a checklist of the skills covered in those materials. A 
pattern is revealed in the Frequency of Use columns at grade levels 
where projects with the same PIP can be compared: Each site shows a 
different set of most frequently used materials. In the most extreme 
case, seen in Table E-4b (individualized reading instruction) for the IRIT 
projects, the frequently used materials form virtually nonover lapping sets. 

The tables show that when project lists overlap, they usually over- 
lap on those materials specified in the PIP as core materials. Conse- 
quently, although the core sets of materials differed among sites, we 
have a slight indication that the instructional staff understood the 
core of the curriculum intended by the PIPs. Since the reading materi- 
als used in every lab project did cover the entire range of skills, the 
skills checklist is disappointing as an indicator of the degree to which 
projects implemented the curriculum. However, we know from interviews 
and informal observations during site visits that project staff generally 
understood the skill emphases intended in the PIP, even when they did 
not implement those emphases. 

The readint» skills checklists do not: reveal i^tfiy differential rele- 
vance of the MAT battery to the curricula, except perhaps for HIT; in 
HIT Lexington the more advanced levels of the MAT, requiring reading com- 
prehension, would be fairly irrelevant to the curriculum. A skills anal- 
ysis performed on the curriculum materials in the absence of information 
about levels, and about lessons at which students were actually placed 
within them, was coo weak to reveal relevance to emphases in the MAT. 
The skills checklist proved disappointing because from our interviews 
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and class observations wo had the strong impression (1) that although 
the materials being used concained lessons on reading comprehension 
skills, such as those emphasized on MAT Primary II through Advanced 
tests, teachers placed little emphasis on these higher-level reading 
skills and (2) that Ccw students were studying anything but remedial 
phonics, decoding, structural analysis^ and vocabulary. 

Table E-1 shows considerable variation among most -used reading mate- 
rials, but shows that all Catch-Up sites used the Random House Criterion 
Reading kit as one of their cere materials, Tiiis diagnostic^ skills- 
testing kit was keyed to lessons in only a few of tiie materials speci- 
fied in the Hardware/Software Packef:, and the Project Director's Manual 
did not clearly describe how teachers should ind^^'x their teaching materi- 
als to the skill areas in the Random House kit. Teachers in every site 
except Galax attempted to choose materials that covered the skill areas 
in the Random House series and attempted to index the lessons^ games, 
books, or worksheets accordingly. 

The Catch-Up PIP described an eclectic approach to reading instruc- 
tion and provided little guidance on what to teach or on how to teach 
particular skills. Rather, the PIP encouraged teachers to exercise judg- 
ment in choice of materials and suggested methods for providing frequent 
success experiences and praise for the students. This pedagogical phi- 
losophy was understood by project staffs. For teaching reading and re- 
mediating reading difficulties, the PIP recommended selecting whac ap- 
pears to be best lor each child from among a variety of materials and 
equipment; emphasis on phonics was inferred from the specification of 
the Random House Criterion Reading Kit. 

In Wayne City^ the Random House series was taken so seriously for 
determining skill coverage essential to Catch-Up that staff created a 
"Core File" (shown as a most frequently used material in Table E-1. 
This file contained individual worksheets and pages removed from a vari- 
ety of published series and kits and filed according to the skills index 
specified by Random House. The project director at Wayne City purchased 
the Fountain V;illey diagnos tic /pros criptive kit !:o aid teachers in iden- 
tifying materials that presented lessons keyed to the skill deficiency 
areas. As shown in Table E-1, Catch-Up sites also had a major portion 
of their most frequently used mathematics materials in common with the 
PIP -recommended core . 

Except for G lover svi 1 le, where staff were implementing the Wisconsin 
IGE program rather thian the Conquest program^ the Conquest projects 
shown in Table E-2 used the P IP-recommended core materials more fre- 
quently than did other PIP projects. Each used some materials focusing 
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on reading coinprclic^nsion skills, but the primary emphasis was on prelim- 
inary reading skills and vocabulary development. Instructional staff in 
Conquest projects understood and implemented the curriculum intended in 
the PIP. 

In HIT (Table E-3), Remedial Reading Drills was used quite fre- 
quently in both Lexington and Olean reading centers • Lexington also em- 
ployed the PIP-specif ied Stories of the Inner City, but Olean did not. 
During the first year RMC reported that Stories of the Inner City was 
mistakenly included in the HIT PIP- (It had been transposed from the 
Conquest PIP.) For the second school year, the Lexington project staff 
made several attempts to get the Sullivan company representative to de- 
liver the materials used in the original HIT site, but were unsuccessful. 
Tiiey continued to use Stories of the Inner City to supplement the heavy 
core curriculum emphasis on phonics drill work. Olean had used the rec- 
ommended Sullivan materials since the beginning of the field test, but 
st.-^ff felt Sullivan was too limited in its focus on phonics and pronun- 
ciation. In this site, staff added more reading comprehension materials 
to tiicir curriculum tlmn the PIP intended. 

Of all the PIPs, IRIT (Table E-4) had what one reading specialist 
called "the most well-balanced reading curriculum." The original PIP 
explicitly covered decoding, vocabulary and comprehension, and individual- 
ized reading for comprehension and enjoyment. More than the other two lab 
programs (especially more than Conquest), IRIT relied on the teachers to 
select the materials they thought would best teach the skills to their 
students . 

The IRIT PIP designated 51 core materials and 71 supplementary mate- 
rials. The sheer quantity of core materials indicated that the intent 
was to specify a very loose base from which teachers x^ere free to vary. 
Materials listed covered a variety of vocabulary and phonics materials, 
reading comprehension ^^7orkbopks, storybooks for fun reading, and audio- 
visual materials for motivation and enjoyment. In general, descriptive 
information of recommended materials . and their use was not provided by 
the PIP. In addition^, many of the recommended materials covered the 
same area of instruction (e.g., several readers were recommended); that 
teaciiers were to choose from among the materials was only implied. This 
confusion has been eliminated by the revised IRIT PIP, which clearly 
specifies that teacliers choose one or two materials from each category 
of recommended materials. The revised IRIT also includes more detailed 
explanations of skill coverage in each category. The recommendation in 
the Hardware/Software Packet (now called Materials /Equipment Catalogue) 
that teachers choose materials different from those used in the regular 
classroom remains in the revised PIP. 
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Although the IRIT L" ieid- tes t projects did not: use many of the speci 
fied materials^ the spirit of the program seems to have been carried out 
All three sites adequately covered the areas of phonics, vocabulary and 
comprehension^ and individualized reading. The projects placed slightly 
more emphasis on spelling, compreliension*^ vocabulary development, and 
phonics than did the PIP. Bloomington and Schenectady used more 
criterion-referenced skills testing, and Oklahoma City added typing as 
an application of language arts. In the main^ however^ the siime general 
skill areas were emphasized. 

In summary^ several points can be made about the congruence of the 
core curricula in tl'.e projects and in the PIPs and the relevance of 
those core curricula to the }\AT : 

• Ttie skills checklists suggested that the MAT was perfectly 
a()nroDr iate for aieasurinR the specif ied -and -used curricu- 
lum. We know from other observations^ however, that the 

curriculum materials contained a vv^hole ran^e of skills (e.g., 
readinc comprehension skills) that few students covered. 
Moreover^ unless one knows which lessons are being studied 
within the curriculum materials^ one does not have much 
information about skill emphases. Knowledge of the spe- 
cific skills that individual students have been studying 
is necessary for determining whether the MAT is a valid 
measure of proj ec t curriculum. 

• None of the PIPs --except PTR and^ to a large extent^ HIT-- 
contained information about exactly which materials were 
responsible (along with effective teaching) for the effec- 
tiveness of the exemplary program in its original site. 
This is not a criticism of the PIPs as a communication de- 
vice. PIPs carried themessage (from the original project 

staff and RMC analysts) that the use of exactly the same 
materials was not necessary. Such a message, however, was 
devastating to the PIPs as a replication device that prom- 
i^3'^J to prescribe the conditions required for achievement 
gainc . 

• Under the Title III grants^ v;hich required that ad'nerence 
to the PIPs be monitored^ project staffs sought to under- 
stand the skill emphases and other aspects of curricular 
philosophy comiriunicated in the PIPs. They generally under- 
stood the curricular philosophy but could infer skill em- 
phases more easily when materials specifications were 
clear^ core and supplementary materials were distinguished^ 
and the recommended list v;as limited to a few materials. 
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Most project staffs decided to endorse and implement the 
skill emphases they inferred. Some (e.g.^ Gloversville 
Conquest and Oloan HIT) decided to reject or modify them. 
When open-ended recommendations were made in the PIPs 
(e.g., to acquire materials that would enable them to 
teach individual students better)^ staff often searched 
for more guidance and structure than the original PIPs 
provided . 

.Our observations and interviews with teachers indicated that they 
were likely to use already-familiar materials if given a choice between 
those and others that were designer! to accomplish approximately the same 
objectives. In addi::ion, when teachers were unfamiliar v/ith curriculum 
materials., they were more inclined to use them if they could be given 
convincing reasons for doing so, were given information on how to use 
them, or, best of all, had time to familiarize themselves with the mate- 
rials during pre-program^ in-service training. These observations re- 
sulted in revisions in the Matcrials/'Equipment Catalogue and some addi- 
tions in a new Training Manual. The revised PIPs for the three lab pro- 
grams include more information on the purposes^ advantages^ and disadvan- 
tages of each material. The R-:3 and lab PIPs, however, still allow 
teachers the freedom to choose among the many materials that were 
present at the original project si5:e. 

Our observations ttxoX teachers used already- familiar materials made 
us question whether the PIP projects were innovations at each site^ 
whether the PIPs had influenced teachers to adopt new materials sug- 
gested in the packages, and whether the PIP project curricula at each 
site were different from the regular school curricula. 



5.5 The Regular Classroom Curriculum 

Although the evaluation was designed to enable us to attribute ef- 
fects to the PIP projects, achievement gains clearly cannot be attrib- 
uted solely to the projects. For all projects except R-3, students 
spent most of their instructional time during the year in their regular 
classrooms; thus, posttest scores at the end of the school, year were af- 
fected by both the special project and the regular instructional program. 
An important consideration for our interpretation of any achievement 
gains was the nature of the alternative curriculum — that is, the curric- 
ulum in the non-PIP classrooms from which PIP students were sent. 



Although we v/anted to know about the reading and math skills the 
students were learning in their classrooms, we did not have the massive 



resources for a ihorougli study of the regular curriculum. However^ to 
achieve a rough idea of the alternative explanation for changes in 
achievement, regular classroom teacliers or school principals were asked 
to list their core reading and matli materials. We thus achieved a gen- 
eral idea of the skills emphasized in regular classrooms and attempted 
to examine the rclationsliip of these skills to the PIP curricula. 

As discussed, the PIP programs, except for R-3, were supplementary 
programs. Catch-Up, Conquest, and PTR were daily "pull-out" programs 
that were not supposed to replace participation in regular classroom 
reading instruc tioi;. LRIT students spent approximately three liours in 
the IRIT classrooms, during whicli 'tliey could liave missed their regular 
reading instruction, but they participated in the IRIT program for only 
one cycle (i.e., a 10- or Il-weck period). 

A review of non-PIP reading and math curricula revealed that^ al- 
though the materials varied^ the skills included in the PIP curriculum 
at each site were also covered in the regular classroom. A single basal 
reader was central to almost every elenientary level^ non-PIP reading cur- 
riculum; these basal readers were sometimes also used in the PIP classes 
to ensure that the skills learned in the projects were appropriate to 
the performance required in the regular classroom, Gloversville Con- 
quest was an exception. Tliere the curricula in both t'he regular class- 
room and the lab were the same; they were based on the Individually 
Guided Education program developed at the Wisconsin R&D center. 

Tlie pedagogical philosophy in reading was the same from classroom 
to lab in most sites. Both the PIP projects and the regular classroom 
empha*Jized phonics. Variation between project and regular classes may 
have occurred in tlie sequence and manner] in which new skills were intro- 
duced, with somewhat greater emphasis on reading compreliension in the 
basal readers and an increased degree of attention to individual needs 
given in the projects. It seemed likely that some of the unspecified 
materials being used by project teachers \^Jeve those materials that they 
liad used previously in the regular classroom. 

The curricula bt the HIT and R-3 programs were different from the 
other four PIP projects and require additional comments I'lie HIT curric- 
ulum, as outlined earli^j^r, centered upon remedial plionics. HIT students 
did little work on the reading comprehension skills being practiced by 
(though not specifically taught to) otlier students at tlieir grade level. 
Reading achievement gains could probably be attributed to the HIT curric- 
ulum except tliat phonics skills were not tested on the test batteries for 
students in grades 7-9. It is more likely tliat practice in reading com- 
prehension In th.eir regular classrooms v;t)uld have helped PIP participants 
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most on the MAT. The R-3 program integrated the reading, math, and so- 
cial studies skills oE an entire grade level at a participating school. 
Teachers were expected to intersperse the recommended R-3 instructional 
techniques among more f ami liar methods^ thus encouraging students to re- 
spond more enthusiastically to the curriculum and to achieve greater aca- 
demic gains* Technically therefore^ any achievement gains could be at-- 
tributed to the curriculum of the R-3 projects because the entire school 
year experience of the students was their R-3 program. On the other hand, 
we do not know whether students would have performed differently in the 
absence of the R-3 program. 

Clearly then^ except for HIT^ projects did not have a curriculum 
that was significantly different from the regular classroom curriculum. 
Although the IRIT and Catch-Up PIPs recommended using materials and 
equipment that were different from those used in the classroom, they did 
not intend differences in skill coverage; they meant only to provide les- 
son variety. Thus^ for all but HIT, it would be difficult to separate 
the effectiveness of the PIP from that of the regular curriculum. 

Failure of PIP students to make gains when their project teachers 
used the recommended materials and followed the PIP instructional style 
(assuming the test is appropriate to the skills covered) would indicate 
that the PIP had failed. Success of PIP students in achieving gains, on 
the other hand, would have to be attributed to both the PIP and the reg- 
ular curriculum because we cannot separate PIP effects from the effects 
of the regular classes which PIPs are designed to supplement. 



5 . 7 Detailed Correspondence; Between the MAT and Fourth 
and Eifihth Grade Curriculum 

The analysis reported in the preceding sections are at a fairly 
nross level oC detail. In our view a much better analysis would be to 
match OLir data on lesson plans with the MAT item scores. Surely a con- 
vincing analysis of PIP project failure would be that items failed in 
the fall and known to be covered during instruction, were not passed in 
the spring. As already noted, one could not show success if the items 
were passed unless a way was found to argue that the regular school 
curriculvim did not cover the items. 

The outcome of an ai-gument based on known-to-be covered items seem 
to us to be so compelling that we attempted an analysis which matched 
curriculum information and item scores on our fourth and eightli grade 
children, wiiere wo gave the' same test fall and spring. 
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The resuLt.s were rcwnrciin); in that we found so few items and 
children for analysis, we felt confident that we had partially explained 
why the original project results were not replicated: the MM did not 
test what these compensa ti.u*y reading teachers were doing. 

The results were disappointing in tliat we did not feel that we 
could report any formal analyses on such a thin data base. However, 
we report the steps of oui* procedure in Appendix G, for those who may 
wish to try an analysis at this level of detail. 

Our curriculum analyses have shown that the MAT was not especially 
relevant to the PIP project materials. The question naturally arises, 
were the original validating tests any better? The next section 
addresses this issue. 



5 • 8 Tests Used to Validate Qri^^inal Programs Compared 
with the PIP Curriculum and the MAT 

Ostensibly, we set out to evaluate the effectiveness of the PIP 
using the same criteria that were usod to validate the original programs. 
In Section 3, we presented the results of a norm-referenced analysis 
using RMC's original criteria for effectiveness. We did not, however, 
give tlie same tests a.s were used for validation in the original sites. 
Thus, we felt it important to compare the MAT with those tests. Table 
5-7 shows the validating tests used, by program and grade. We wished 
to determine whether the validating tests were aligned more closely than 
was thie MAT with the PIP-speci f ied curriculum. We could then determine 
whether differences in the way they aligned with the PIP curriculum 
would account for the difference in test gains. 

The skills tested must be discussed at a more general level than 
waj tlie congruence elaborated in Appendix G, because a fine-grained 
analysis was not conducted. We have no records documenting what cur- 
riculum was used at originating projects, only what was specified by the 
PIPh. Consequently, our argument will turn on other judgments as well 
as extension from the analyses given previously. 

To compare the validating tests with the PIP curriculum^ we used 
the general level skills on the MT as referenco. points. For the Primer 
MAT, for example;, we isolated the following five .^ikills: (1) matching 
beginning sounds with pir^jres^ (2) matching ending sounds with pictures, 
(3) matching beginning sounds with letters^, (4) matching ending sounds 
V7ith letters, and (5) matching spoken words with written words. Skills 
of a similar quality v;ere developed for the validating tests. Whenever 
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Table 5-7 



TESTS USED TO VALIDATE EFFECTIVENESS OF ORIGINAL PROGRAMS 



PIP 



Validatinfe Test 



Catch-Up 



Metropolitan Achievement Test (grades 1-3) 
Comprehensive Test of Basic Skills (grades 4-6) 



Conquest 



Gates-MacGlnitie (grades 1-3) 

California Achievement Test, 1957 (grades 4-6) 



HIT 



Wide Range Achievement Test (grades 6-8) 



IRIT 



California Achievement Test, 1970 (grade 3) 



PTR 



Gates-MacGinitie (grade 1) 



R-3 



Comprehensive Test of Basic Skills (grade 8) 



possible, we used the same list of skills for the validating tests as v;c 
had used for the >L\T ; however, when some items on the validating test 
could not be described by these skills, we added the necessary new cp.te- 
gories of skills, 

\^en all skills were defined, it became apparent that many of the 
skills listed for the MAT were almost the same as those listed for the 
validating test^i. Ti^e skills were then reviewed to see which ones, if 
learned by a student, would allow him to answer more items correctly. 
The percentage of icems devoted to each skill was computed for both To- 
^al Reading and Total Math. For each test, this computation was made by 
dividing the number of items in each reading or math skill category by 
the number of items devoted to reading or math skills in all of the sub- 
tests . 

Finally, che skills were reviewed for coverage in the PIP curricu- 
I'jir.. We were somewhat limited in this analysis because we had not exam- 
ined all of the specified materials, only the specified materials that 
were used at the field-test sites. However, we were generally familiar 
v/lrh the nature of the specified materials and the skills they covered. 
Although we lacked conclusive evidence, wo were confident that some of 
the skills covered by some of the tcr.r.s were not covered by the PIP 
curriculum materials. These skills were marked as a no ' or a not 
certain,"" dcpendiivj. on hov confident we v;erc that they were not covered. 
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Table 5-8 



COMPARISON Oh- I'USTTEST TOTAL READING fONTENT BETWEEN THE MAT USED IN THE PH' 
tVAl.UATK>N AND TESTS USED IN EVALUATION OF ORIGINATING PROGRAMS 



Tfrit /CdDI ent 



Total Keadins 

Listening tor Snuudi 

Hatch beijinninK saun<J with picture 
Match t>iidinK uound with picture 
Match beginntoK sound with letci^r 
Match ending iound with letter 
y;atch spoken with written word 

Woi d Knowledgt^ 

Match word with picture 
Match written word wit^ writtm wnrl 
Oct in i t ion 
Oppositcb 

Reading 

Match sentence with picrurt* 

Match story (2-5 sentences ) wl th picture 

Single word jniiwer to riddle 

Single paragraph sturies witii ques,i>ns 

Literal 

Inferent ia 1 

Main idea 
Recognitiop ot letier ujmes 
Match word with picture 



Grades I and 2 



Grade 1 



KAT Primer 
Form F (T - 72) 
PTR CCjnton onlv) 



Percent of 
To t a 1 Te 9 c 



1") 

7 



HIP- 
Spec 1 t led 



Yes 
Ves 
Yes 
Yes 
Yes 



Y-; 'i 
Yes 



MAT Primary I 
f-'orm F (T • 77) 
PTR (Dalla!i only) 
Catch-Up. t Conqu est 



Percent ot 
Total Test 



lOOZ 



( 12J 



PIP- 
Spec i f led 



Gates-MacGini t ie* 
J-'orm 1 , Primary A 
(T - 82) 
Conquest , PTR 



Percent of 
Total Test 



Yph 



27 



PIP- 
Spec i f ied 



Yes 
Yes 



MAT Primary 11 
Form F (T • 84) 
Catch-Up, T ConqueaL 



Percent of 
Total Test 



20; 

28 
(23) 
( 5) 



37 
(25) 
( 6) 
( A) 



PIP- 
Spec i f ied 



Yes 
Yes 



Cates-MacGinit ie* 
Korm I, Primary B 
(T - 82) 

Conquest 



Percent of 
Total Test 



59i 



5 
3b 



PIP- 
Spec i f led 



Yfs 
Yes 



^l!r-?T'r''''^*"'M~" ''r' vocabulary and comprehension into « Total Reading .core at the Primary A level. Nevertheless, percents for 

Total Readme .re displayed because the Dissemination and Review Panel does not state which .core, were used to determine the exemplary pro.rL. 

Key: T - numD«-r ot test items; NC - not certain. Examination of fourth .,nd eighth grade curricula in sample classrooms provided some anchors trnm whieh 

.te thorsUn: r hi'h H '•^"-•^^^^^ ■ " — ^ ^-^^ ^ -t other grade level.', the notation "NC" is used t" 

indi^.te tnose skills tor which theru was no pvtdence. 



Teat uaed in ev.il u.it ion of i«riginat mu progrnm." 
^MAT also used for .'Vjluatinn of or i fiiiiit ing prt»Kr.Tn. 
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Tabl« 5-8 (Contlmitid) 



i and U (Cont lud«'«I ) 



fpst /tonti-nt 



Grade U 










Cal L f ornia 


MAI lilem«,'nt ary 


Cal L formal 


Test of Basic Skills* 


Kurm F (T - 95) 


Ach ievptnen t Teat * 


Konn Q, 


Level 2 


Carch-Up, 


Conquest , 


(4,5.6- 


-195/) 


( . 5 , 6 j 


(T - 85) 


IRlTt 


(T - 120) 


[Conquest 


Catch 


-Up* 


P«rcont of 


I'lP- 


Percent oi 


PIP- 


Percent ot 


PIP- 


Tot.il Tt?st 


Spec it i 


Total Test 


Speci f led 


Total Test 


Spcc i f ied 


OUi 




lOOX 




lOOZ 






































Yes 






ii: 












10 


NC 










11 


No 










10 


No 








Yes 


















53 






Ves 


9 


Yes 


16 


Yes 




YC9 


6 


Yes 


11 


Yes 




Yes 




Yes 


6 


Yes 




No 






1 


No 






3 


Yes 










i; 


Yes 










17 


No 










u 


No 
















NC 










13 


No 


7 


VOii 












Y.jii 












Yes 












Yf;o 










8 


YfS 












NC 






17 


NC 




NC 


7 


NC 








rjc 






23 


NC 


1 




10 


NC 








Yt>« 












Vt'H 












VpB 












Y*>fi 










h 














Ye. 






10 






Y<»s 












Yes 


1 J 


Yea 










U 


NC 


30 


NC 




Y.JH 












Y^'fl 












V.'s 












Y.'s 












'l t' s 






U 


Yrs 


1 




1 ; 


Y..9 


iU 





Total R«>>idine 

Word KnowledRO 

Match W(.ird with pictiire 
Match written word with written word 
Dot i ni t ion 

M.iCfh w,ird in phrase with tivnonym 
oetwr.il vnrabulary 
!jp»»c » a I ized Words 
Mathemat ics 
Sr i enc e 

Soc i <i 1 »rudi es 
Opposires 
M/iCch i^pi^ken wnrd with written word 

Readinj C umpr Chen s ion 
Stories with "iiiestionrt 
Lit.T.a 

I n £ t? r e n t 
Mam idva 

Word m con'^ext 
i!eniience of ev»»nts 

I I ovd i ni'. direction 
Keleri'Mce skills 

Or t*.an i/.ir I nn .it topics 

M i see 1 laneous 

A! phahet i £ I Mi' , mdrx, and 

t.ihli* -J cunt.-nts 
l\u*t r v 

r'<i,-««t.lpe ti';:t 11 jt stor;t."S 

Vt<«. .Tbu lar V riniv fby ^Tide Iev»'l ) 
. ^ 

?. J. J 
3. J- 1. 



. J - 
1 ii. ; - i 
1 (J . ■» - i 1 
i 1 . 1 -: 1 
li . )-. J 
1 J. :-n 
i K ; - 1 



(ic»* leni't h (in uot d * 



iv.i I 1 I 



V 1. .ihu I tr V I .ir..'- 
'Jh- fiHittti ^raiJ- 
.\ 'A :hr ei- .jdt a 
' • ; 1 } t •! ' : «r ! 
. »-v- is HI t f" 



• r 1 i? i nn t 1 iif; pr r an . 

!..r t lur f li iT.iih; IM r \'\\ 



third /T .1*1 ''r H w»>rt' 



sti'd at tlu 



or 1 c in/(t inR site. 



r tl.irv I-vr-ls r.i?^!»r fh.in the -ih^.^lMt*- level r>| e/irh pnMH.ii/e is djaplnyed f^rt this cOiapariHon. 

\ \\\ ; iIKS wre franod hy thi? raiU"'' ni th«' MAI pflfisai^e/!. While th" MAT ranj^eii chanj^e frnm 
' I- 1 >■ Ti'iir If V ) t "I ttie fifth i\ni <i\%l\\ i^riHtes ( i n t e i ned i t e ) , the R/ime CitlS was -idmi n i ■ tered 
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Table 5-8 (Continued) 
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Table 5-8 (Concluded) 
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The content of the validating tests is compared with the content 
of the MAT in Table 5-8 (for posttest Total Reading) and in Table 5-9 
(for posttest Total Math). These tables show the skills covered by each 
test and the percentage of items devoted to each skill. Skills are 
grouped according to the subtests in the MT. When added^ the "percent 
of total test" for all skills within a subtest will equal the **percent 
of total test" for that subtest. The total number of reading items (or 
of math items) is listed as "T" at the top of each column. In the fol- 
lowing sections, we discuss the results of our analysis of Tables 5-3 
and 5-9. 

Catch-Up - -Catch-Up validation x^as based on the same levels of the 
MAT for grades 1-3 as were used in our analysis, but the validation for 
grades 4-6 was based on the California Test of Basic Skills (CTBS). Be- 
cause the tests for the first three grades were the same, they were 
equally appropriate and the Lest scores should be comparable. 

The CTBS and the MAT have a somewhat different format for word 
knowledge items. The CTBS tends to be a little easier because it gives 
the target word in a phrase that helps clarify the meaning. This format 
also matches the PIP curriculum better than does the MAT because most ma 
terials required the student to determine the meaning from the passage.*^ 

The format for the CTBS items on reading comprehension is similar 
to chat of the MAT except that the CTBS includes a section on poetry and 
the MAT has twice as many inferential questions. Grades 4-6 at the orig 
inating site all took the same level of the CTBS, whereas the students 
at the field-test projects switched from the Elementary MAT to the Inter 
mediate MAT in the fifth grade. At Grade 4, the level of the MAT pas- 
sages is closer to the fourth grade materials than the CTBS, which is 
substantially more complex. In gradcis 5 and 6 the positions are re- 
versed, and the CTBS pas.sages more closely match the curriculum. The 
confounding factor for grades 5 and 6 is the poetry section on the CTBS, 
and for grade 4 is the larger number of inferential questions. Even so, 
this does not outweigh the differences In the passage levels. 

The CTBS Total Math subtest emphasizes computation more than does 
cither the Elementary or the Intennediate MAT. The CTBS and the Inter- 
mediate M/\T emphasize whole numbers and fractions; the Elementary MAT em 
phasizes whole number operations. As stated earlier, the Catch-Up cur- 
riculum focused most heavily on computation skills- It would seem that 



fiov/ever, students are rarely tested or. this Ilfy in tiie exerci.:ey 
tiiat we examined Tor the fourth and eighth grades. 
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the CTBS is more relevant: than either MAT battery^ but in grade 4 (Ele- 
mentary MVr); where most students were working with whole numbers^ the 
extra items on fractions in the CTBS could mean that it is the less rele- 
vant test. 

Th.e heavy cnphasir* in the M^M on both concepts and problem solving 
was not reflected in Catch-Up curriculum. The CTBS seems to be closer 
to the PIP curriculum because it does not emphasize either subject^ and 
the items that do deal with concepts and problem solving use the tradi- 
tional math style. 

Overall, the CTBS seems to be more appropriate to Catch-Up' s PIP- 
specified curriculum. Assuming that this similarity would help students 
to answer more of the test items correctly^ we feel that (except for 
grade- 4) the project might have proved more effective in terms of both 
reading and math scores if the validating test had been used. 

Conques t - -The validation for Conquest V7as based on three levels of 

the Catt's-MacG i n i L i c lor grades 1.-3. The replicating sites were tested 
on L h r e c i e vols o l I h o MAT . 

The Gates-MacG initio has a larger percent of word knowledge items 
than do any of the MAT 'batteries . The Primary II and Elementary (grades 
2 and 3) batteries of the MAT emphasize matching written words with 
written words, while the comparable levels of the Gates-MacGinitie con- 
tinue to use some items that simply match words with pictures. The 
matchinv; of a picture with a word was a little more common in the 
ConquesL curriculum than the matching of written words. Thus, for 
grades 2 and 3 wi> believe the Gates -MacGini Lie is more closely aligned 
with the PIP-spec i f Il.o and used curriculum. The word knowledge sections 
ol bouh tests seem to be equally appropriate for grade 1. 

A similar comparison about pictures can be made relative to the 
readiiig cur.iprehension items for grades 1 and 2. ScnnL Gates-MacGinitie 
items simply require matching a story of tv/o to five sentences with a 
picture, while the MAT items compel the student to read a single para- 
graph sUory and ansv/er Mome questions about it. By grade 3 both tLists 
have stories v.dth questions, but the Gates-MacGinitie has only literal 
and Inferential qut-.stions, v/hercas the MivT also has main- idea and word- 
in -con text Items. The Conquest curriculum contained a fair amount of 
read ing CL.niipreheus ion materia 1 s v;l th a pic tun' f c rai^t , which br inf-'.s it 
0 L o s e r 1 1-- l : • e C / i t . - M a cG i. n i f i e i r, r a J e s ] a ; i d 2 - In g c a d e 3 t h e added 
t y pes o f c ^"^s [1 Cv. 1 1 questions (; a the MAT a a in :n a ke s the Gates- 
MacGinitie vl't niovi' i' IP-rc levant test. 
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Overall, in grades 1-3 the Gates-MacG initie appears to be the 
better test for the Conquest reading curriculum. 

All students in grades 4-6 at the originating site were tested on 
the same level of the 1957 California Achievement Test (CAT). At the 
replicating projects, we administered the Elementary MAT to grade 4 and 
the Intermediate MAT to grades 5 and 6. The difference in the dates of 
the two tests, the L957 CAT and the 1.970 MAT, will influence the results 
of this comparison because the older test reflects a somewhat different 
content emphasis in testing. 

The MAT stresses word knowledge skills more than does the CAT, but 
the CAT includes more specialized words from mathematics, science, and 
social studies. Conquest did not emphasize any specialized words, and 
so the MAT seems to be the more appropriate test. 

Although a larger portion of Total Reading content is devoted to 
reading in the CAT, only 20% of the test deals with reading stories and 
answering comprehension questions. About 387p of. the Total Reading iuems 
cover skills like following directions, reference skills, and organiza- 
tion of a topic; although some Conquest materials covered following 
directions, we found none that dealt with the other two skills. 

Overall, even though we found Conquest materials that followed the 
reading comprehension format ot both the MAT and the CAT, we feel the 
CAT to be the less relevant test for reading skills because it includes 
skills not covered in the curriculum. For this reason, students should 
have scored higher on the >L\T. 

HIT- -HIT was evaluated by the same level of the Wide Range Achieve- 
ment Test (WRr\T) for grades 6-8. The Intermediate MAT was used to test 
grade 6 in our analysis; the Advanced MAT was used for grades / and 8. 

All reading items on the WRAT deal with reading a list of words 
aloud. The test lias no word knowledge questions and no stories with com- 
prehension questions. The HIT curriculum emphasized phonics and oral 
drill work. The curriculum offered litLLe reading comprehension material 
and did not cover most of the vocabulary in the Advanced MAT Word Knowl- 
edge subtest. The WRi\T is much more closely aligned wit!, thu emphasis 
in the HIT I'-riadirg. curriculmiu 

Ail ma::h items on the WRj\T are computation problems. Of tne basic 
operation items, 42% entail working with fractions, and 33% x^/it.h whole 
numbers; the MAT places less emphasis on fractions. Because the WRAT ex- 
cludes math concepts and problem solving and because the HIT curriculum 
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cmphasizu-J computia L ions using both whole numbers and tractions but did 
not emphasize either math concepts or problem solving^ we believe the 
I'mAT more closely parallels the focus of the HIT project. 

In addition, the Wll^VL" is substantially shorter than the MAT bat- 
teries. The WRAT has 118 items across both reading and math and takes 
30 minutes to administer. The MAT has 210 items and takes 125 minutes 
to administer. These are the only two tests for which we feel that the 
difference in the number of items and the testing time would affect test 
scores. The MVr is substantially more difficult and more taxing of stu« 
dents' abilities than is the Wlli\T , 

Overall,, the WRAT is more appropriate for HIT curriculum. With the 
added feature of its short lengthy we believe that the field-test proj- 
ect would have seemed more effective if HIT students had been tested on 
the WRAT. 



1RIT--IRIT was originally validated on the 1970 CAT for only grade 3 
The third grade stadents in our study were given the Elementary MAT. We 
find it diffictilt to say what skills the IRIT project was trying to empha 
size because it specified so many materials. We do know that IRIT ' s cur 
rlculum was divided into three sections: phonics, vocabulary and compre- 
hension, and individualized reading. Phonics is not covered by either 
test, but the other tv;o areas are covered by both. 

Although the MAT fiives the word knowledge skills a larger portion of 
the Total Reading score, the CAT has two formats for these items. Half 
of the items require matching a spoken word with a written word; the 
other half, wl.Lch uses the same format as the MAT, requires matching a 
wrLucen word with a similar written word. We are unable to say if one,. 
of those formats received more emphasis in the IRIT curriculum. Rela- 
tive to word knowledge, both tests appear to be equally appropriate for 
the IRIT curriculum. 

The Mi\T features more items on reading comprehension than does the 
CAT. The M\T ' s o.ir.phasis is on inferential questions^ the CAT*s on lit- 
eral qu-:>L,oi'K The IRIT :ua\:^'i,A^ r.ivvoled sl.ijrlir rr-ference for lit- 
eral quci^-f xom; . CAT passage features uul only are lower than MAT fea- 
tures, but also are probably at c: level more commonly encountered by 
third grade students. As does its earlier edition^ the 1970 CAT tests 
some skills that do not relate as directly to reading and are not obvi- 
ously covered in the IRIT curriculum. The CAT has items that are 
missing on the MAT, such as reading a table of contents and an index. 
Kvon tiiou^th flic p'lssages on the MAT are more difficult tiian the ones in 
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the IRIT materials, the percentage of CAT items dealing with skills not 
covered by IRIT appears to make the CAT the less relevant reading compre- 
hension test. 

Overall, however, tiie CAT is an easier test so, even though the CAT 
has some skills not covered in the IRIT curriculum, students at the 
field-test sites might have done better if they had taken the CAT, 

PTR--The Gates-MacGinitie was used to validate the originating PTR 
site. In our evaluation^ we tested grade I in Canton with the MAT Primer 
and grade 1 in Dallas with the Primary I. 

None of the skills in the Listening for Sounds subtest in the Primer 
are covered on the Gates-MacGinitie, but they were excensively covered 
in the PTR curriculum materials. 

The word knowledge items in the Primer were included in the Reading 
subtest. The Gates-MacGinitie puts a heavier emphasis on word knowledge 
than either of the MAT batteries. The PTR curriculum included lessons 
on word knov^led.c;e that are similar to items on both tests. 

The composition of the Reading subtest is different for all three 
tests (Gates-KacGinitie, Primer, and Primary I). The only item common 
to all is that of matching a sentence with a picture. The other items 
on the Primer examine recognition of letter names. The other two sec- 
tions of the Primary I deal with riddles and with reading a single para- 
graph and answering comprehension questions. The other section of the 
Gates->'acGinite requires matching a picture with a story of two or three 
sentences. The PTR curriculum did not cover riddles, and the single 
paragraph with questions and the matching of a picture with a multiple- 
sentence story are found only at the more advanced levels. The curric- 
ulum placed tremendous emphasis on recognition of letter names. The 
single-sentence picture match was also covered in the curriculum, but not 
as heavily. Of the three teslr,, the Primer is probably the best reading 
test for PTR because it empb.asi/:es the 3a:na skills as ':he curriculum. 
The Gates-MacG i ni cie would be the next, best test, and the Primary I 
the least appropriate. 

Overall, the Primer appears to be the best test of * the PTR program 
because it covers the skills emphasized ia the curriculum and includes 
more of the .skills on wliich PTR focused. The Gates-MacGinitie is le.^s 
appropriate, and the Primary T, the least appropriate. 
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R-3--The test for validating the original R-3 project was the CTBS . 
For our evaluation, R-3 students were given the Advanced MAT. 

Word knowledge skills are emphasized equally on t!ie MAT and the 
CTBS, but the formal: on the CTBS is different. This format, which gives 
the target word i.n a phrase that 'nelps clarify the meaning, more closely 
matches the curriculum materials. 

The reading comprcliension items receive approximately equal empha- 
sis on both tests, with CTBS having additional sections on miscellaneous 
skills and on understanding rules and poetry. The MAT passages are more 
difficult than the ones in the CTBS, and the MAT places greater emphasis 
on word-in-context and inferential items • We were unable to find any 
evidence that the R-3 materials covered skills like understanding rules 
or poetry. The more difficult passages in the MAT were not reflected in 
the curriculum, with the possible exception of the most advanced levels. 
Although we could not determine from the curriculum how the MAT empiiasis 
on inferential questions would have affected the students' scores, the 
lack of word-in-couLcxt skills in the curriculum could have had a nega- 
tive effect. 

The CTBS stresses math computation more than does the >L\T, espe- 
cially basic operations using fractions. Altliough both tests emphasize 
math concepts, the MAT has many more items cn modern math. The MAT also 
has a much heavier emphasis or- problem solving. However, the R-3 curric- 
ulum co^^ered such :i tremendous variety of skills that it is difficult to 
say whit skills would be considered core to this program. Because of 
thi<; variety in math skills and formats, one test cannot be designated 
che mo^e appropriate for this program. 

Overall, we fuel that R-3 students could have done equally well on 
both tests. For p\-.nmnlc^ in the Reading subtest che increased diffi- 
culty of the MAT stories is balanced by the additional skills required 
on the CTBS. Relative to math, lack of knowledge about what was core to 
the program prevents distinguishing between tests. 

5 . 9 Cone lusiom; 

In 10 of die 1,9 PlP/gradc combinations^, the validating test pro- 
vided a closer match v;ith curriculum and a better chance of showing proj- 
ect ef fectivunes^^ . In five PIP/grade combinations, the MAT was the more 
cippropriate Lose, In the remaining foar combinations, neither the pre- 
test nor th.L ],rst test could be rated sabstant laL ly better; three of 
these cr,i-hi f.i!:i(^ns arc for C.'atch-Un, er iil^s 1-3, v;liere the MAT was used 
for both jirc- and po.s t - tcs t i. ng . 



In our analyses, we attempted to verify that there was reason to be- 
lieve that successful projects would increase MAT scores. Such verifica- 
tion was dependent on showing that the MAT was congruent with the curric- 
ulum specified by the PIP and used with students. 

From a rather gross analysis of MAT skills^ the MAT appeared rele- 
vant to PIP curricula^ even though PIP projects exhibited considerable 
diversity in their selection of teaching materials (e.g., IRIT projects, 
which showed almost no overlap in materials). The diversity of curric- 
ula may not be counter to RMC*s expectations, since they packaged the 
'^programs** of the originating sites even when such programs had no 
consistent instructional methods or specific curriculum materials. How- 
ever, we regard this diversity as an unsatisfactory outcome for pre- 
scriptive packages that promised to cause the same achievement effects. 
At the same level of analysis, we found that (except tor R-3) the PIP 
project curricula W3re not episentlally different from each other or from 
the recular school curricula. That is, th3 PIPs did not transport funda- 
mentally innovative projects, although they did establish working proj- 
ects. This is not a limitation of packaging, but rather a consequence 
of what was selected for packaging. 

From our fairly detailed analysis of the relationsliip of the MAT to 
fourth .-^nd eighth grade curricula and our less detailed analysis of the 
relevance of the validating tests to all curricula ob:;erved, we con- 
cluded that for most PIP projects the validating test would have been 
more responsive to the curricula. Our analysis of fourth and eighth 
grade curricula showed that only the MVI Math Computation subtest seemed 
particularly relevant to PIP objectives. In our norm-referenced results 
(Section 3), the math scores stand out as an area of project success com- 
pared with MAT Total Reading; this confirms that some of what was taught 
was learn id .:?.nd, giv.?n a relevant tesc, the eVs-^iJlaacor can detect that 
learning had occurred. We are not surprised that scores were low in the 
areas not covered by the PIP because, according to our limited informa- 
.tion, these areas were not covered by regular classroom curricula either. 

We have found no reason Lo believe that MAT scores should be greatly 
increased by participation in PIP projects. PIP curricula are not inno- 
vative, but were supportive of the regular curriculum, so that credit for 
the limited successes of the PIP projects must be shared with the regu- 
lar classes. 

The main variables manipulate.! by PIP projects to improve MAT 
scores over scores expected from regular instruction appear to have 
been classroom manH.^enienL and Lower student/teacher ratios. Pir-'luduced 
curricula did nor tench the child anything relevant to the MAT that he 
could not learn frr-n^ y^llu-c sources. Most important, the MAT was not 
found to bo pa rt Leu L.i r Ly relevant for assessing the achievement impact 
of PIPs. 
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6 'HIE EFFECT OF IMPLEMENTATION ON ACHIEVEMENT 



6 . I Intiroduct ion 

Because the MAT was nor particularly relevant to the curricula of 
PIP projects, we had no specific reason to expect large gains in MAT 
scores. Nevertheless, hecause these scores were the only measure we 
had for PIP effects on student achievement, we present some formal 
analyses relating them to teach.er implementation and responsiveness 
(descrihed in Section 4). Preparatory to these analyses, we discuss 
some simulations done to guide our choice for the metric of the depen- 
dent variable. We conclude that the MAT standard score metric is 
defective and is therefore inappropriate for evaluative purposes. 

Our main conclusion is that tho formal analyses reported below do 
not suppor': the claim thac PIP iniplementation alone produces large gains 
on our measurement of achievement. Teacher responsiveness is more often 
e f fee t Lve . 

In this evaluation, generally as.^in others of its scope, no sampling 
of project teaciier:;. students, or locations was possible. As a conse- 
quence, the empirical justification for the usual inferential statistical 
techniques is not present. The basis for our analyses is rr^t that of 
inferential statistics, but that of curve fitting and "least squares" 
descriptive statistics. Consequently, we will not report our results 
according to the canons of statistical decision theory. The problem 
with the inferential framawork in this study is that with no sampling 
scheme we have no basis for claiming that the probability statements 
associated with hypothesis tests have any em.pirical significance." 

6 . 2 Definition of Reer ession Model for Teacher Implementation 
: md Responsiveness Ratings 

In Section 4 we described how we rated teachers on two factors: 
implementation of project and responsiveness. These variables were 
nominally scored according to the sclieme shown in Table 6-1. The ratings 
were tlien converted to tlirc:- "dummv^ variables" p er project , as follows: 



lve uo ne t adopt a Hayosl.^n point o t: view for the corresponding reason: 
We have no sa t is lac tm-y posterior distributions. 
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Well Implemented 

Well Good or 

Implemented Responsiveness Good Responsiveness 



dc") (^W/G*) 

1 1 

0 I 

0 I 

0 1 1 

0 0 0 

0 0 0 

0 1 I 

0 0 0 

9 0 0 0 



Rating (^w') 

1 I 

2 1 

3 1 
4 
5 
6 
7 
8 



Table 6-1 



CLASSIFICATION SCHKME FOR PROJECT TEACHERS 
DURING OBSERVATION 



Teacher's Implementation 
of Pro: f^c i 


Teacher' s 
Responsiveness 


Good 


So-So 


Bad 


Wei 1 impleiiu r. ^ - J 
So-so implemented 
Poor I V implemented 


1 


2 


-» - 


4 


r 


6 




8 


9 



Most of the anal 'scs report- i '-. low entail only ly and I^; when this 
caused singularities, 1^/^, was used. The distribution of teachers on these 
variables may be inferred Erom Table 4-5 .n S ■ tion 4. 

Our basic descriptive model is a bivariaL- regression equation: 

(6-1) 

5k Ijlm i.im i.im ijkm 
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m = 1,M Indexes projects. 

^ i>2 Indexes fall and spring observations, respect ive] \ . 

1,J^ Indexes teachers within projects, when there are 

J teachers. 

m 

■'■ '-T.j^ Indexes students of the jm^^ teacher, when there 
are T. students for the jm^^"^ teacher. 

If the jm^^"^ teacher had a well-implemented projeci 
during our site visits. 



Otherxv^ise . 

If the jm^^ teacher was responsive during our site 
V is i t o . 

Crherxv/ise. 



i jm 



Lj Im 
I 

I Jin 



i j km 



If the ijm^^ student was male. 
Otherwise. 

Is an indicator variable for the student's race. 
The exact specification of this \;ariable depends 
on the ethnic * distribution of each project. See 
Appendix D, which shows the independent variabl:^s 
used in equations for the regression analysis. 

Is the ijm^^ student's age in the fall. 

Is indicator variable for each student. The 
p:irrmet> r associated with this variable is shown 
for coirpleteness sake; we will not estimate it. 

Represents the error for student ijm at time k. 



Our evidence for FIP effectiveness would be that 



1 2m 1 Im 



( 



for a large percentage of our PIP and grade combinations. When this in- 
equality holds, our model asserts that, given the responsiveness of the 
teachers and the values of the other variables at a PIP and grade, being 
a student of a teacher with a better implemented project makes a greater 
impact in the spring than in the fall. If our inequality does not hold, 
we have no positive association between degree of Implementation and out 
comes and no evidence that Implementing the PIP well is assoc-^ated with 
increased val-os of the dependent variables. 



213 



The model (Hq. 6-1) suffers from the do f ecu s "tha t the norm-referenced 
procedure (Section 2.3.2) attempted unsuccessfully to overcome. The model 
does not tell us what would occur if no project were in place, nor does 
it say how far from zero the inequality (Eq. 6-2) must be to be educa- 
tionally significant. A problem portlier expression (Eq. 6-1 nor Eq. 6-2) 
addresses 's how many successer*lccording to the model would imply that 
the PLPs ar?. successful. 



6 . 3 Selec fion of Metric for the Outcome Variable 

As a result of the content analvses presented in Section 5, we know 
that the MAT items are not highly relevant to the PIP curricula, and, as 
a result the analvses described in Section 2, we know that the longi- 
tudinal validity of the MAT norms is questionable. The implication is 
that the MAT standard scores may not be an appropriate metric for the 
analvbis of PIP project achievement outcomes. 

The "grade effects" on the Reading subteh:t discussed in Section 3.7.1 
can be interpreted as further evidence of this. In that section we found 
that the percentile of first and fourth grade Reading subtest averages 
declined as a function of time over a wide variety of project types, loca- 
tions, and student body characteristics. If these declines are artifacts 
(i.e., if the declines do not reflect some defect common to all projects 
at these grades), obviously the MAT standard score metric should be 
abandoned . 

ConsL-qucntlv, we decided to investigate whether there might not. 
artifacts in the MAT standard scores that would cause apparent declin-js. 
Our investigation was conducted by means of the simulations described j ii 
the next sent i(^n. 



D . 



'] . 1 Simulation of the Norm-Referenced Analvses 

for r. he MAT Rending Subtest--Grades 3, 4, and 5 



We liad observed in the PIP data a definite trend toward gains in 
percentile of p.-oject averages in the thir<l and fifth grades and losses 
iu the fourth. Obviously, if the M/\T norms were valid in 1970, the de- 
clines could bo the reflect Lon of some developmental factor that came 
into plav in children ten vears old in 1976, but that was not present 
in V-^70. Alternat ivolv, if tlie norms are currently valid, these declines 
ini-ht mean serious defect in all fourth grade PIP curricula. A third 
alternative is that the fourth grade MAT norms are not, and never were, 
valid. Evvdcuco f^r thi^ view can be obtained from the Anclior Tust Study 
(1974). AL^cho lourlh -ivide Llie Anclior Study percentiles for rending 
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between the 60th and 2nd (see Table 28 in Anchor Test Study "Equivalence 
and Norm Tables for Selected Readiag Achievement Tests" (1974). 

For our simulations, we created raw score di3tributioii5; in which the 
"effect" of a program was to increase by a fixed proportion, B, the number 
of items answered correctly." The question is, will programs that are 
equally effective in this sense, be equally effective in the sense of the 
norm-referenced analysis for reading that we used for the PIP projects, 
no matter what member of the MAT battery is used. 

It is not n^ctrssary that programs that are equally effective on a 
MAT raw score metric be equally effective on the MAT standard score 
metric if the projects are tested using different members of the MAT 
batcery. However, if they i: not, the inequality is evidence that the 
standard score transformati s not of the same form at each MAT level. 
If the transformations are not of the same form, the underlying distri- 
butions are not comparable. This may mean that the underlying traits 
being measured are different. 

Our simulation generated 3000 pseudorandom variables, Pj^, i = 1 to 3000, 
distributed as the Beta, with parameters o^^ and Ct^, so that the mean of 
the distribution of the simulated fall standard scores, calculated as 
below, was about what we had observed in the PIP study. 

The were converted to Reading subtest standard scores for each 
grade by the following formula: 

= [P + G(l - p) ] , (6-3) 

where R^ is the fall raw score, is the number of items on the MAT used 
in the fall, and G is a gviessing parameter, 

= PNp + B(N^ - PNp) , . (6-^4) 

where Ng is the- number of items on the M/.T used in the spring, and Np is 
the number of items on the spring test that are parallel with items in the 
fall test. B is the "effect" of the program, as compared with no relevant 
education at all. 

R = C -f G(N - C ) , (6-5) 

s s s s * 

where R^ is the spring raw score and N^^ is the number of items on the 
spring test. 



These simulations xs'ere pro;::;rammod by George Byrd, Pat McCall, and Roy 
Sutton of SRI, usine IMSf.'s Heta random number generation. 
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The '3000 raw score pairs, and R,., were converted to standard 
scores through the MAT raw-to-standard score tables as follows: 







Reading 


Crade 


Test i.nt; 


Subtest Table 


3 


Fall 


Primary II 


3 


Spring 


Elementary 


4 


Fail 


Elementary 




Spring 


Elemencary 


5 


Fall 


Elementary 


5 


Spring 


Intevmedlate 



The 3000 fall and spring standard scores were used to, generatie 100 
norm-referenced analyses calculated on 30 ^^servations apiece. The same 
3000 observations were used at each grade that had the same value of a'^^ 
and p., 30(k - n -f 1 i ''^ 30k, were used in the k^^ analysis, 

k = U 100. 

The results of these simulations are presented In Tables 6-2 and 6-3. 
Each table shows the percentage of the norm-referenced analyses at each 
combination of p ^v^, and grade, which resulted in the various decisions 
on the achievement of normal growth and criterion growth. The statistics 
In. the table refer to averages in the norm-referenced analyses. 

Table 6-2 shows the results of our simulation for selected values 
of B, when G = 0. X^[hen B = 0, this table shows that the norm-referenced 
analyses are not subject to grade effects. However, the mean gain over 
expected Is much higher In the fourth grade than in the third and fifth 
grades. When B = 0.1, the norm-referenced analysis begins to show grade 
effects, with 517c of the analyses iji the fourth grade confirming normal 
growth, while none confirms normal growth at the other grades. At 
B = 0.2, 100% of the fourth grade analyses confirm normal growth, while 
11% of the third grade and none of the fifth grade analyses show it. 
Thus, for data like those found In the PIPs, our simulation shows that 
when'c; - 0 the tests are d I f f eren t la 1 Iv sensitive to B, the proportion 
of Items "due to program." In the simrilation present^^^^ 
" one would characterize' the four th grade a s ea sler than the third or the 
fifth grade, given the BVs we have used In our model. 

Other simulaL-ions confirmed the results when grades were compared 
usln^^ the B^s shown on Tables 6-2 and 6-3 with the following ^^^s common 
to all grades: ^ - ^ = 2.0; - 2.0, = 4.0: = 4.0, cv^ = 2.0. 
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Tfiblc 6-2 



mmim of simulated norm-referenced m\m fop, hat wmw subtest, 

KI.T!1 THE INDICATED DECISIONS FOR THE ACIIIEVDlENT OF CRITElilON GROWTH 
AND NORMAL GROWTH: C -- 0.0 







Grade 3* 




Grade 4 


Grade 5* 






Alpha 


1 = 2.0 




Alpha 


1, ^ 2,0 


Alpha 1 = 2.0 


. . _. 




Alpha 2 = 2.0 




Alpha 


2 ^ 4.0 


Alpha 2-2.0 


B ■' 0,00 ^ItH'ts I'l'i ( (*r ion iTt^uth 




01; No 


lOOZ; U OZ 


U'S 


OZ; No lOOZ; U OZ 


Yes 0/.; No 100/,; H 0/ 


Hi't»ts mml gu'wth 


Yes 


OZ; No 


lOOZ, U OZ 


Yes 


OZ; No lOOZ; U CZ 


Yes OZ; No lOOZ; U OZ 


Fall score 


Mean 




, SD 2.17 


Mean 


43.40 


SD 2.94 


Mean 60,79; SD 3.00 


Spring score 


Mean 


31.17 


; SD 1.67 


Mean 


48,40 


SD 2.94 


Mean 43,81; SD 1,62 


Expected score 


Mean 


46.97 


, SD 1.98 


Mean 


53.09 


SD 2.68 


Mean 66.42; SD 2.86 


Coin uver hU 


Mean 


-14.31 


; SD 0.70 


Mean 


0.00 


SD 0.00 


Mean -16.99; SD 1.44 


Cain over expected 


Mean 


-15.80 


,SD 0.65 


Mean 


- 4.69 


SD 0.36 


Mean -22.61; SD 1.32 


I) - 0.10 Hppr^ rr i Tori ["n y^nL'tfi 

I' V'lU III^L? WILL "It. 11. tiUt^lM 


1 L'3 


OZ; No lOOZ; U OZ 


les 


OZ; No lOOZ; 0 OZ 


Yes OZ: No lOOZ; tl CZ 


ileets normal growth 


Yes 


OZ; No 


98Z; U 2Z 


Yes 


5).Z; No OZ; U 49Z 


Yes OZ; No lOOZ; h OZ 


Fall score 


Mean 


45.48 


SD 2.17 


Mean 


48.40 


SD 2,% 


Mean 60.79; SD 3.00 


55pring score 


Mean 


41.81 


SD 1.28 


^lean 


54.45 


SD 2.25 


Mean 55.01; SD 1,18 


hptoclcd score 


Nean 


46.57 


SD 1.98 


Mean 


53.09 


SD 2.68 


Mean 66.42; SD 2.86 


Gain over fall 


Moan 


- 3.67 


SD 1.02 


Mean 


6.06 


SD 0.73 


Mean - 5.78; SD 1.86 


Gain over expected 


Moan 


- 5.16 


SD 0.90 


Mean 


1.36 


SD0.55 


Mean -11,40; SD 1.73 


B f 0.20 Ht'ets criterion j;rowth 


Yes 


IZ; No 


12Z; U 87Z 


Yes 


lOZ; No OZ; U 90Z 


Yes 07' Nn 7f!7- 11 Vi 


Meets normal growth 


Yes 71"/.; No 


OZ; U 29Z 


Yes 1 


OOZ; No OZ; U OZ 


Yes OZ; No 16Z; il 84Z 


Fall score 


Moan 


45.48 


SD 2.17 


Mean 


48.40 


SD 2.94 


Mean 60.79; SD 3.00 


Spring score 


Mean 


50.21 


SD 0,77 


Mean 


59.43 


SD 1.75 


Mean 64.25; SD 0.85 


Exptected score 


Mean 


46.97 


SD 1.98 


Mean 


53.09 


SD 2.68 


Moan 66.42; SD 2.86 


Cain >ver fall 


Moan 


4.73 


SD 1.43 


Mean 


11.03 


SD 1.25 


Mean 3.46; SD 2.18 


G.-)in over expected 


Mean 


3.25 


SD 1.26 


Mean 


6.34 


SD 1.02 


Moan - 2.16; SD 2,04 



Note; r ' unknown. 



Kesiilti in these columns arc based on the ,same 3000 observations disLributed as Beta with aj - Ct7 = 2.0. 
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IMi'I'liiniON (IF Siyil'LATED MRM-RtFEIiENCED ANALYSES FOR HAT READING SUBTEST, 
;-'lT!l lllE INDICATED DECISIONS FDR lllE ACHIEVEMENT OF CRITHUOX CROWni 
AND NORMAL GROWTH; C 0,25 







Grade 3 










n 1 r 

Grade 5 






Alpha 1 ^ 2.0 






Alpha 1 - 0.1 




Alpha 1 = 1.5 






Alphn 2 - /t.O 






Alpha 2 = 1.0 




Alpha 2 = 5.0 


1: • fi \]\\ 


ilrt'L^ 1 1 LU l U'l! \;rOwUl 


'fes 07.; No 997.; il 


17, 


leij 


u/o, iNo luu/o; u 


A'/ 


ICS \}k\ m iUU/o; u u/« 




iU't.'lS n.UillJ 1 L;ll'VrlIl 


Ves 997.; No OZ; 0 


IZ 








VniJ 1 17' Mn n7' II flQ7 




Fj'.i score 


He.in 47.68; SD 0.88 




\\m 


49.19; SD 1,90 




Mean 56,99; SD 1.26 




Spring score 


Mean 50.98; SD 0.46 




Mean 


49.19; SD 1.90 




Mean 63,58; SD 0.52 




Expected score 


Mean 48.74; SD 0.86 




Mean 


53. 7o; SD 1,77 




Mean oc.vl; SD 1. 13 




Gain over fall 


Moan 3,29; SD 0.49 




Mean 


0.00, SD 0.00 




Mean fc,59; SD 0.77 




Gain over expected 


Mean 2.23; SD 0.47 




Mean 


- 4,57; SD0.30 




Mean 0.67; SD 0.64 


W - iMU 


Meets criterion growili 


Ves 977.; No 07.; U 


3Z 


Yes 


01; No 1007.; U 


U/o 


Yes 0J/9; No U/fli U J//o 




Meets normal srpwtti 


Vcs lOOZ; No OZ; L' 07. 


Yes 


971; No OZ; 1) 


ih 


ICS 1UU <; m U/o; U U/o 




F:ill score 


Mean 47.68; SU 0.88 




Mean 


49.19; SD 1.90 




Mean 56,99; SD 1.26 




.'pririf; score 


Mean 55,40; SD 0.37 




Mean 


55,55; SD 1,56 




Mean 69.33; SD0.33 




lixpected score 


Mean 48.74; 0.86 




Mean 


53.76; SD 1,7? 




Mean b2.91; SD 1. U 




t'li'i ever fall 


Mean 7.71; SL 0.56 




Mean 


6.36; SD 0.43 




Mean 12.33; SD 0.96 


i 


Gain ever expected 


Moan 6.65; SD 0,55 




\\m 


1,80; SD 0,41 




Mean 6.41; SD 0.83 


B ■■ 0.20 


Meets erU'erion gnivtli 


I'es 1'^''"; No 07.; 0 


OZ 


Yes 


42Z; No OZ;, U 


587. 


Yos lOOZ; No O'l; D OZ 




Meets iior:ii,il I'.rowth 


Yoi 1 ; No 07,; U 


OZ 


Ye,s 1001; No CI; 11 




Yes lOOZ; No OZ; U 0°, 




I'll 11 score 


Mean 47.68; SD 0.88 




Mean 


49.19; 3D 1.9& 




Mean 56.99; SD 1.26 




iiprinp, score 


Moan 59.17; SD 0.31 




f-lean 


59.33; SD 1.44 




Mean 73,53; SD 0.27 




Expected score 


Mean 48.74; SD 0.86 




Mean 


53.76; SD 1.77 




Mean 62.91; SD 1.13 




G.iin over j'.iU 


Mean 11.68; SD 0.61 






10.13; SD 0,53 




Mean 16.54; SD 1.01 




Clin over expected 


Mean 10.62; SD J.:i9 




)\m 


5.57; SD 0.47 




Mean 10.62; SD 0.89 



Xot'c: II - iinl'.nou'n. 



Table 6-3 shows the results of our simulation for the same values 
of B discussed above. In this tabb», •.• = 0.25, and the Q^Vs have been 
changed from those shown in Table to make the fall average scores 

about the same as were observed in the PIP studv. 

In Table 6-3, the results for the fourth g/ado are largely unchanged. 
The results for the third and fifth grades are dramatically different, 
however. When we simulated uo guessing, the fourth ^rade test appeared 
ea-^ier in terms of results for a fixed value of B. Table 6-3 indicates 
■ uit with simulated guessing (G = 0.25) the fourth grade test is now the 
most difficult. In fact, some analyses confirm normal growth in the third 
and fifth ^^.^ades, even when B = 0, 

Ovei-all, these grade effects are similar to the grade effects noted 
in Section 3.7, They indicate that the norm-referenced analysis as ap- 
plied to MAT reading data gives different results for programs that are 
equally effective (as measured by B in Eq. 6-4). Which reading test is 
easier depends on the size of gain that is allowed. If the values we used 
for Rg are increased in Eq. 6-5 by the second ter^i, the fourth grade is 
harder than either the third or the fifth. If we do not increase them, 
the fourth grade is easier. It is important to note that this result, 
however, is relative because the fourth grade standard scores are nearly 
stable. It is the third and fifth grades that are sensitive to the changes 
we have simulated. 

These findings show that the items selected for the various MAT 
batteries were not stich that the batteries have comparable distributions 
of item difficulties? in their rc.specti\/e norm groups. Of course, there 
is no requireme.^t ::at t there be coaicarable norm group distributions of 
item difficulties. The Thurstone techniques that are the analytic founda- 
tions of the MAT standard scores may be applied no matter what the 
raw score distributions. Indeed, the standard score transformation rv.^y 
be. viewed as correcting for the fact that the noin"' group distributions 
differ and that^ therefore, equal raw scores on different batteries do 
not represent equal values on an assumed (unobservab le) underlying 
normally distributed skill continuum. 

The usefulness of this correction depends on the reasonableness of 
the assumption of the existence of the unobservable skill continuum and 
the reasonableness of the presumed distribution cf children's values on 
this continuum. 

We feel that the three reading subtests--th'at of the Primary II, • 
the Elementary, and the Ad vnnced- -are • sn f f ic lent ly dissimilar to justify 
interpreting the differing distributions of item difficulties as evidence 
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that these jsubtests are not measurinp the same skills. The Primary II 
Reading subtest entails much lower' level reading skills than does the 
Elementary.' The Intermediate has a' much higher vocabulary level than 
does the Elementary, and it also has much longer paragraphs. It is our 
view that phese factors, the points raised in Section 2, and the grade 
effects dis'c\issed in this section and in Section 3.7 warrant abandoning 
the MAT standard scores and using the raw scores instead. 



In the' 



next section, we discuss the dependent variable used for our 



raw scare apalyses. 



6.32 Definition of the Dependent Variable ^"^ijkm^ 

The decision to abandon the standard score metric caused two problems. 
The first was that the proportion of correct responses is not a ^dependent 
measure with constant: variance, as the standard scores (in theory) nearly 
are. Consequently, we could not^ expect our least squares procedures to 
work well on this measure. The second was that, where we did not give 
the same pre- and post-test, we had to rescore the MAT using only parallel 
items. The items we judged parallel are shown in Appendix E. Table 6-4 
shows the numbp.r of parallel items we used, by grade, and the total num- 
ber of items possible. We selected items based on our judgment of what 
was parallel, not based on the MAT publisher's intentions. Unfortunately, 
we did not find many parallel items where we did not administer identical 
pre- and post-tests. 

We imposed an additional restriction on the items at grades 4 and 8, 
where we did administer the same pre- and post-tests. As reported in ... 
Appendix G, we had determined a subset of MAT items that we were reasonably 
sure had been covered at those grades. Consequently, we excluded other 
items, leaving only those shown in Table G-5 for analysis. 

To correct for variability in the variance of students' scores, we 
weighted each score inversely to its standard deviation. Thus, we let 



V . , in '\q. 6-1 he. 

I 1 Kin 



T(P.., ) 



1 jm ijkm 



ijkm' . /P. (1 - P. ) 

\/ ijkm iji^ 

whore P^^m-j^ is the proportion of items that student ijm answered correctly 
at casting k, out of the N^. possible. \^en P = 0, we set T(P) to its 
^...allest possible finite value for N. When P = 1 , we set ..T.(P) to its 
largest possible finite value for N, 
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Table 6-4 



NUMBER OF ITEMS ANALYZED AND NUMBER OF POSSIBLE ITKMS, BY GRADE 





Test 


Total Reading 


To ta 1 


"Mo f-V* 


Numb e r 
Parallel 


Numb e r 
Possible 


Number 

ra ..^ i ie .1 


Number 
Po ss ib le 


Fa 1 1 


Spring 


1 


Primer 


Primary I 


• 6 


72* 


24 


34 


1 (Canton) 


Primer 


Primer 


72 


72 


34 


34 


9 


Primary I 


Primary II 


10 


77 


J J 


62 


3 


Primary II 


Elementary 


13 


84 


46 


108 


4 


Elementary 


Elementary 


78 


95 


58 


115 


5 


Elementary 


Intermedia te 


12 


95 


41 


115 


6 


Elementary 


Intermedia te 


12 


95 


41 


115 


7 


Intermedia te 


Advanced 


11 


95 


46 


115 


8 


Advanced 


Advanced 


73 


95 


38 


115 


8 (R-3) 


Advanced 


Advanced 


73 


95 


105 


115 



The Word Analysis subtest in the MAT Primary I corresponds to the Listen- 
ing for Sounds subtest in the MAT Primer. The Reading subtest in the 



Primer corresponds to both the Word Knowledge and Reading subtests in the 
Primary I, but the correspondence of the latter subtests with the Reading 
subtest was too difficult to analyze. 



The transformation has the effect of cV];inging proportions according 
to the following tabulation: 



P T(P)/VN 

0.1 0.33 

0.2 0.50 

0.3 0.65 

0.4 0.82 

0.5 1.00 

0.6 1.22 

0.7 1.53 

0.8 2.00 

0.9 3.00 
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The LransEomed values increase faster Cor larger proportions than 
for smaLlor, making it somewhat easier to detect small differences in 
large proportions than to detect the same differences in small propor- 
tions* Thus, whether the transformation works in favor of noting dif- 
ferences, or against, depenui on the size of the proportions involved. 

Any artifacts introduced by the transformation are secondary to the 
artifacts that are introduced by having so few items to analyze after 
the elimination of those that are not parallel pre and post. v^e were 
surprised to discover that the tests had so few parallel items, given 
that the tests can supposedly be used in or out of level. 

In the next sections, we report the results of our analyses of the 
:est results; the items analyzed were selected and transformed as dis- 
cussed in this section. 

6.4 Results of the Analyses of the Unadjusted Transformed Raw Scores 

Because of the small number of items analyzed, it is inappropriate 
to regard the analyses of the transformed rav; scores as definitive. 
These are, however, the best data on student achievement we have. 

Table 6-5 shows the uaaijusted "effects" of the responsiveness and 
implementation variables described in Section 6.2."''^ Tabled are the average 
gains in students' transformed raw scores for teachers that site visitors 
scored as having good responsiveness, and the average gains for the 
students of those scored as having bad responsiveness. Similarly, tabula- 
tions are shown for implementation--that is, for students of teachers 
judged to have well or poorly implemented projects. When we could not 
reasonably separate the implementation and responsiveness ratings, we 
combined them as discussed in Section 6.2. These results are labeled as 
"good/well" or "bad/poor." 

Our two main conclusions are that: 

• About 807o of the time, students of teachers identified as 
liavinp; good responsiveness generally showed higher average 
gains than did students of teachers identified as not 
responsive . 

• About 507o of the time, students of teachers identified as 
havin • veil-implemented projects had larger average gains 
than students of teachers having poorly Implemented 

proje'^is . 

'Raw data files were prepared by John Rollin; analyses were executed by 
George Black and Pat McCall. 
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Thus, res pons i vonoSb as defined in jur observations seems a fairly 
good predi'^tor of which teachers' students will show larger average 
gains on our transformed metric, while our implementation rating does 
not do as wel 1 . 

Uur results must be viewed with caution .owever, because of the 
i?.aeven distributions of raimbers of students in the various categories 
of teachers, and, as already notei, because of the generally small number 
of items being analyzed. A further caution is that our "bad" and "poor" 
teacher categories also included the "so-so" teachers (see Section 4,7 
for definitions of these terms) . 

Table 6-6 shows the corresponding tabulations for the transformed 
MAT raw scores for math. Here, neither of our observation variables is 
very successful. However, at the fourth and eighth grades, where we 
have the best evidence that our dependent variables are relevant, an 
association exists between observers* judgments of good responsiveness 
and gains. For the other grades, we find it difficult to say that, 
responsive teachers are successful in teaching MAT reading items but 
not successful in^teaching MAT math items. This difficulty is especially 
obvious because so few items and students were available for some responsive- 
ness/ nonresponsiveness comparisons . 

Even without the problems discussed, we '-'ould be reluctant to attri- 
bute the evident success of "responsive" teachers just to their rated 
responsiveness; there are competing explanations that we have not in- 
vestigated. One such explanation is that a bias exists in the age, race, 
and sex distribution of students, a. bias that works in favor of the 
responsive teachers and against those having well-implemented projects 
(although thesiftwo factors arc not independent). 

The bivariate rej^'; lession model described in Section 6.2, if it fits 
the observations well, will permit a ccjmparison that eliminates these 
biases. The interpretation of this model is presen'ct d in the next section, 

6 . 5 Results oL" the Analyses of the Adjusted Transformed ^aw Scores 

To implement the model discussed in Section 6.2 using the reading 
dependent variables d^-'scribed in Section 6.3, we ran ten step'^.^ise bivariate 
regressions: one for eacli j^rade , 1 through 8; a separate run for PTR 
Canton, Mississippi, grade one; and a sep-^rate run for R-3, grade S. 
For mathematics- we ran the same regressions, but excluded projects, where 
mathematics is not part of the c'» rricu lu:u . At HIT and IRIT, "good 
responsiveness" and "well Implemented" ore completely confounded. In 
Catch-Up, grades 2 and '3, these variables are nearly confounded, and we 

223 

9 - ' 
'J t..' 

o 

ERIC 



combined them into a single variable as discussed in Section 6.2. The 
complete specifications of the regression equations for reading and math 
are shown in Appendix D. 

Because we can calculate thc^ difference B^^^^^ - ^^^^^^^^^^ 

culating the parameters H^^^,, our procedure was to minimi/;e the variance 
of the res idua Is : 

1 j km L J m 1 j km 

Our first concern is for the models' fit to the data. 

('..5.1 G.K Hjness of Fit for Fall and Spring Regression Runs 

. To assess the goodness of fit of r ; ol at each grade, we cal- 
culated a residual, J^^^i^ni' ^^^^ each c 

R. = B. . + ^ 

L j km L j m 1 j km 

.,;liere ^j^jj^i^i the error for individual ijm at time k and where B^^^ is 
•1 unestimated parameter associated with each Individual. This parameter 

..A introduced because tlie l^^ji^^ ^^V ^^^t be independent of the P|^j2m* 

:'o^ each bivariate regression equation, goodness of fit was assessed by 
vamining the joi[it distribution of the ^i^i^ ^nd l^^yi^ ^ Lu ictlon 
:f our implementation rating. As expected, the R^jini ^^^^ ofUMi liighly 
•)sitively correlated with the Rj^^2m* According to our model, there 
hould be no large negative correlations and there were none for either 

reading (^r math, for either implementation status. 

rahle b-l sh^v.v's the standard devLalions of the residuals for the 
fall and sprin.^' tr-msformed r-ading rav; scores. If the equations fit 
v.'ll al e ich Imn lem-.-ntat inn status, Lhe standard deviation of the 
res Idua Is i .> ibi .u t i . 

t;ra::.- S ' and grade I Canton PTK sluiw rather large standard 
deviations of reuling residuals, wliicli indicates a poor fit to those 
data. Overall, readin,; d.ita for tlie other grades are fit reason. ti)ly 

1 1 , hut the fi* wiiiiin graile <.lt)e^^ vary by implementation status, tail 
or s[u"ini:. rt- ^; r-. ss i ( ei vri'iatiiMi, and V IW At the second and third g.rades , 
(.:on(jiiest ' ii: is - I igh t 1 v wc^ rs-.' l haii Ca t c h -Up ' s , hut tlie rt'Verse is 
tru(? at tiu- i'.u:rlh ,:rad-. In I iU: 1' tlu data of teachers with poor iiupb.'- 
!iienrati(Vi t.a.i I- fit. Intter, while ii^ (;oinju..'.-t the data of sucli Le-achers 
1 re I i I :u>:Ar\:U . ' r . 
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Table 6-5 



UNADJUSTED "EFFECTS" OF TEACHER RESPONSIVENESS AND IMPLEMENTATION 
ON STUDENT GAINS ON THE TRANSFOEMED RAW SCOPES; READING 





Catch-Up 


CoriC 


uest 


HIT 


IRIT 


PTR 


R-3 




R''^sponsive 


ImpleTnented 


Responsive 


IiDplenented 


Responsive 


Implemented 


Responsive 


Implemented 


Responsive 


Implemented 


Responsive 


Implemented 


Grade 


Good 1 Bad 


Well 


Poorly_ 


Good 


Bad 


Veil 


Poorly 


Good 


Bad 


yell 


Poorly 


Good I Bad 


Hell 


Poorly 


Good 


Bad 


Well 


Poorly 


Good 


Bad 


Well 1 Poorly 


Grade 1 
Hean 
N 


2.512 2.693 
8 8 


2.119 
5 


2.823 
11 


















1 






0.731 
17 


1.567 
37 


1.202 
29 


1.422 
25 








Grade 1 (Canton) 
Mean 
N 






























6,0)7 
18 


6,853 
45 


-- 


6.631 
63 








Grade 2 

Mean 


Good/Well 
0.812 
6 


Bad/Poor 
).048 
9 


3.056 
26 


0,985 
31 


1,889 
29 


1,972 
28 






























Grade 3 
Hean 
f] 


0.2^3 

J 


0. 


481 
8 


2.029 
13 


i.ia 

21 


1,596 
24 


1,244 
10 










Good/Well 
13 


Bad /Poor 
1.612 
52 
















Grade 4 
Hean 
N 


\M 1.069 
z^ in 


1.232 
21 


1.685 

11 


28 


1 i^AO 

/] 


n 


38 










1,871 
28 


-0.3?] 






\ 










Grade ] M 
Mean ^ 


1.534 • 0.505 
^ 23 20 


m] 

21 


0.614 
22 


1.164 
19 


1,335 
14 


0.683 

11 


1,514 
22 






























Grade 6 
Mean 
N 


0.828 -0.035 


0.935 

14 


0.086 
9 


2.397 

18 


2,034 
15 


1.256 
9 


2,598 
24 


Good/Well 
0,581 
28 


Bad/Poor 
0,050 ■ 
12 






















Grade 1 

Hean 
















0,977 
8 


0.887 
55 


0.83? 
51 


1.158 
12 






















Grade 9 
Hean 
















Good/Well 
0,940 
52 


Bad/Poor 
0.718 
9 






















Grade 8 
Mean 
N 






































2.01-. 
288 


1.222 
228 


l.o64 
- 516 



Note; "Eitfcts" ccupar.ilil. t':^ Eq. (>■'! .irt the lilft'creiices between well jnd poor, or between juod M hi. Hean = average fall-spring gain in students' transforwd raw scores; li ■- number of students. 
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Table 6-6 



UfiADJllSTED "EFFECTS" OF TEACHER RESPONSIVENESS AND IHPLENEKTATION 
ON STL'DENT GAINS ON THE TRANSFORMED RAW SCORES: MATH 







Gate! 


i-Up 




■ HIT 


R- 


3 




Implemented 


Responsive 


Implennjntcd 


Responsive 


Iiaplemented 


Good 


B.id 


yrii 


I'oorlv 


Good 


Bad 


Weil looriy 


Good 


Bad 


Ui^ll 1 Pnnrlv 


Gr.uk' 1 




















0.103 


1.0]0 


1,075 
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N' 


] 


] 


i 


10 










GiMdt' 2 


Good/Well 


Bad] 


'Poor 








■ 


HtMn 




0.654 
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5 






8 










Gr.Kk' i 


















Ho ,111 


■0.1 




0.]05 










N 


7 




16 










Grade 4 


















Mean 


3.401 


2.111 


J. 136 


2.487 










N 


18 


14 


16 


16 










Grade 5 


















Moan 


O.IOB 


1.049 


0.158 


1.565 




■ 






N 


22 


18 


20 


20 










Grade 6 


















Heaii 


0.505 


2.386 


0.491 


1.772 


- 0.490*-. 


■- ■ 0.490* 






N 


18 


7 


15 


10 


23 


23 






Grade 1 


















}\m 










1.370 


0,562 


1.233 






N 










39 


8 


47 






Grade 8 




























1.034* 


-- 1.034"' 






y 










40 


40 






Grade 8 


















Mean 














1.246 


1.217 


4.361 1.225 


V 














269 


291 


1 559 



^iOtoA "i;f[ecLs" cDiiip.ir.iblL' to iiq. 6^2 .ire the Jifforcnces between well and poor, or between good and bad. 
iMiMii ivcraisi' [.ill-spring gain in stiidunts' traiisfoniied raw scores; N ^ nunib(ir of students. 

lu'.sponsivi'n'j>:s i\v\ iiiiploiiientaiion are coniplotely confounded* 



Tiirou^hocit Table 6-7 are Lnstc-nces in whLcli the sLandard deviation 
of tho residuals for studenLs having teachers with poorly implemented 
projects is two to three times that of students having teachers with 
weLl-implemented projects, and vice versa. However, with the exception 
of grades 4 and 8, the standard deviations at all grades are Less than 
3.5 times their expected value, if we exclude Canton PTR. 

Based on oar analysis of. reading residuals, we conclude that our 
model for reading scores is not adequate for R-3 at grade 8 or for PTR 
at Canton. Grade 4 shows generally higher residuals than do the other 
grades, especially in Catch-Up. 'Ihe equations do not tit students in 
the various implementation categories equally well, but tho differences 
are not as gre.u as those seen between grades. 

Tsible 6-8 shows the standard deviations of the residuals for the 
maiii t-qaatioiis. As with reading, eighth grade R-3 stands out as poorly 
fit, wLtii sc'ventli grade HIT also showing large standard deviations. At 
tiie other grades, our models fit the math data fairly well, except at 
sixth grade Catch-Up, where children of teaciiers with poor implementa- 
tions are not Lit as well as are children of teachers who have we 1 1 - 
implemented projects. 

In summary, we have reasonably good fits to both reading and math 
data at all grades except grades I and 8. At grade 8, as discussed in 
Secti(Mi , the MAT was n.ot especially relevant to the PIP curriculum, 
so we will not try to find a better moclel.-; The Canton data have large 
variances, possibly because ol the uneven implementation of the program, 
or because of the small number of items being analyzed. 



In the iu.:xL suetii>n, we discuss tlu- implications of our model for 
the as:^i SsiiienL <n ?} V impact on achlevenient . 

„ . , J. J, s_ioj^i_ Analysis cu" the Effect of PIP Imp lem eiita L ion 

M .] ; lA'!' Trap s Lo rnuul Raw Sc ores 
^ . — — 

(.,.,!• ^eiivr-il ::,.<U! ' vj] • f)- 1 is stieli thai;, if we regress fall-spring 
,,nn-. ui '::e i - p . ul e 1 1 r. variables, (he resulting c ..e T f i c i en t s are the 



di ! i\->--;k'- 



(. o r respond i.nv, fall coe ff ic Lent ard spri nv, eo- 
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Table 6-7 



STAJIDARD miPM OF lli^SIOUALS FOR FALL AND SPRING immB READING RAW SCORES, 
BY PIP, GRAD£^ AND IHPLEMENTATION STATOS 



C:itch"Up 



Conquest 



HIT 



IRIT 



R"3 



Poor 



Grade 



Fall I Spring 



Well 



Pour 



Fall I Spring 



Fali Spring 



Well 



spring 



Poor 



Fall I Spring 



Well 



Poor 



Fall I Spring 



Fall I Spring 



Will 



loor 



Fall i Spring 



Fall [Spring 



Well 



Poor Wi'll 



Fall I Spring 



Fall I Spring , F all I Spr ino 



-Grade 1 
SD 
N 

Grade I (Canton) 
SD 
N 

Giade 2 
SD 
N 

Grade 3 
SD 
N 

Grade 4 
SD 
Jl 

Gnide 5 
SD 
N 

Grade 6 ' 
SD 
N 

Grade 1 
SD 
N 

Grade 8 
SD 
N 

Gr ■ 8 



11 



.131 1.260 
5 



0./7:) 0.369 



{].%8 1.536 



.1.010 

17 

-)■} 



0 



2.653 O.830 

5 

0,8)j 1.643 



2.021 2.831 
29 



n 

21 

1.^31 I'vjl 



1 



1.616 2.276 



1 .085 3,1 



2.814 3,3: i^c::! '.833 

I :, 

I 

1,025 1.12' ;i-2:i) ■ 833 
22 1 

2.215 2.756 .X 0.974 

24 9 



0.905 1.415 
25 

1.0B5 5.998 
63 



1.226* 2,358* 



L207 1 .235'" 



2.710* l^sf 



3.538' 3.612* 



'9* 'J^O* 
1? 



.635 2,151 



.114' '.243*'1J''2* 1.7)[: 



1.340* 



2.121* 2.647* 



.632 



■',548 6. 
516 
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Table 6-8 



STANDARD DEVIATION OF RESIDUALS FOR FALL AND SPRING TRANSFORMED MATH RAW SCORES, 
By PIP, GRADE, AND IMPLEMENTATION STATUS 









' ■ / 


HIT 


R- 


3 




_ P( 


.or 


Weil 


Poor 


Well 


Poor 


Well 


Fall 




Fall Sorinc 


Fall 


Sprini' 


Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


Cradi' 1 


















SD 




1.077 


0.762 1.886 










N 


10 


4 










Grade 1 
















SD 


1.177 


1.292 


0.217 7.524 










N 




9 


4 










Grade 3 
















SD 


1.171 


1.663 


2.817 2.540 










N 


18 


5 










Grade 4 
















SD 


2.143 


4.234 


1.705 3.569 










N 


16 


16 










Grade 5 
















SD 


1.433 


2.065 


2.354 1.968 










N 






20 










Grade 6 
















SD 


3.429 


4.866 


2.551 2.065 










N ' 


10 


15 










Gradi! 7 
















SD 








1.753 


2.487 








N 








47 








Grade 8 
















SD 








3.031 3.922 








N 








40 








Gr.iik' 8 
















SI) 












4.840 


5.925 


0.0 0.0 














559 


1 



^'oti:: N iniiiibur sturlciits. 



for readin^^ and luaLh. B^,^ is ^he "adjusted effect" of a teacher with a 
well-implpmented project, and M.,,^ is the adjusted effect of a responsive . 
teacher. For those cases in which we could not^ estimate Bi^^i ^2m 
separately, we estimated a single coefficient B^n , as discussed in 
Section 6.2; t;hls coeflLCLunt is labeled "well/good" In our tables. 
Computations were done using the stepwise regression procedure of the 
"Statistical Package for the Social Sciences" (Nie et al . ,^1970) ,_so 
that we would get some idea of the relative importance of , or Bj^^^ 
and B^ni, given the other variables in the equations. Ultimately, we 
forced tiie prugrcim lo take all independent variables shown in Eq. 6-2. 

Table b-9 summari/.es the process for gains on our transformed read- 
ing score, and Table 6-10 summari^:L-.s the run for gains on our transformed 
math scores. 

In Table 6-9, the final coefficient o f de term [nation is not par- 
ticularly impressive at any grade, especially at Canton PTR and grade 7 
HIT. To some extent, these results could be anticipated from the results 
of our analyses of residuals. However, the coefficient of determination 
for grad^' 7 HIT is worse than anticipated. 

In this table, we have shown the sign of the final value of the 
coefficient of B^, or of Bl^ and B^^,, as a coefficient to the^increase 
in the coefficient of determination due to 6^, or to and 1^2m* ^^^^ 

Lime it entered tlie equation. In parentheses, we show the order in which 
\v "^Im ''^^^^ ^'^m' entered the stepwise procedure. Generally, the 
implementation and responsiveness parameters do not add mucli to R ; that 
is, the effect of these variables is small. 

Comparing ihe igns ,of the effects shown in Table 6-9 with the 
corr,:^sponding signs caUulated from Table 6-5 shows that adjusting for 
ai;o , race, and se>: .,i 1 uts the picture of responsiveness and implementa- 
tion impacts. Scvc:i .-l-f.'jts that were positive in the table of unadjusted 
effects arc negative ei.t.ects after adjustment. Four effects that were 
nog.ilivc before acljustment are positive afterwards. 

i{.r.,>*vor, '^/.-amiivitioii nf the increase in the coefficient of ueter- 
minauion du.- to inclusion of i ^ , or B,^^ and B^^^^ , siiows that in fivo I'll' 
t;rade eonbinalions the incroaso is less than 0.0013. In these cases, 
knowledge of i::Tple:nen;:ation ::-tatus or teacher responsiveness does not 
ad.d inroniMlion nm. prcs.MiL in. variables in the equation before the 

cntrv of or and i^.^^ . 

If -...t^ iv^iorc cases in vhicli \^ , or Bjj^j and B^^^^, add Lliis liltlr lo 
tiic cuef f i-n(. m: Jc t:cr:ninat i on , in three cases positive unacjusted 



9 3 2 

\ 



2i 



1 (unLwl hi I 1 















(1) 


(1) 








(6) 



i;.r'l 



in i;-) 

t^;.oio .0,008 
i (M (M 

H\OV* 'O.0J5 
C! il) 



iicii;ciH)ii 

'O.OOD +{1.(11)0 

0) 



(.') 



..'I'.l'ily.L J Ji':iPi!:i'i:;l!.'i. . liili' 



(1) 



r 



i:';cRF,:\::r; i:^' m coEFFicinr of mmwmm m mMi differences in nath 









Catch-Op 


HIT 


R-3 










Well 




Well 




Well 


Final 


Cr.ide 
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(2) 
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+0.0)8 
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-0.001 
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+0.001 
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0.027 


8 










M run 








8 




5 
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Note; Nirabcrs in oarentlieses indicate tiie ori'n' entered; si^n is the j^i of the coefficient in the final 
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eEEocts arc iii^v'tiLvr iift.or atljustment and in three cases negative un- 
adjusted effects becomu positive. 

IC we double -count \, as both weLL implemen.cid and responsive, in 
four cases the adjusted etfucL of responsiveness is negative and in ton 
cases it is positive. in seven cases tho adjusted effect of being judged 
well implemented is negativ-e and in four ic is positive. 

Therefore, even after adjusting for age, race, and sex, our general 
conclusions concerniLig tlie unadjusted effects hold. If implementation 
or respon.siveness has any impact, good responsiveness, as we have defined 
it, is associated with small gains on student achievement tests in a 
variety of PIP and grade combinations. 

Table 6-10 shows corresponding statistics for math. Again, the 
equations for grades 7 and 8 stand out as not being very good predictors 
of tlie values of fall-spring gains. In the math data the effect of 
adjustment was to convert one unadjusted effect from positive to negative 
and one from negative to positive. Thus, adjustment for age, race, and 
se:< does not much alter our conclusions concerning the effects of Pli^ 
implementation and teacher responsiveness on our math variable. Howl'^lt, 
the acljustment did increase the number of positive effects ^for responsivu 
ness from four to five and tlie number of negative implementation effects 
from two to three. 

Therefore, in our formal analyses, teacher responsiveness--more 
often than implementation of PIP philor.ophy and speci f ications--is 
associated with increases in the transformed raw scores. 

Caution should txe.cised in generalizing our results to tlie un- 
observed cliildren in oa.: study or to future studies that use procedures 
different from ours. Neve rtlie less , our result on achievement is fairly 
clear: As judged by the nomi- re f e renccd analyses and by the regression 
models just reported, implementation of the PIP philosophy and procedures 
^did not raise MAT scoi::.-; to any imoressive degree. However, the MAT 
content was not especiallv relevant to PIP curricula, and the MAT standar 
scores nuv nol bv wl' M -sii i Led to valid norm-referenced procedures. 



7 SYNOPSIS AND RECOMMENDATIONS 



7 . 1 Sunimary 

la this section we review the outcomes of our evaluation summarizing 
our cone la ions . 

The main issue addressed was the validity of the PIP replication 
principle and the associated norm-referenced analysis. Based on our 
evaluation principles, we conclude that the replication principle was 
false: there is little reason to believe that packages of the type we 
evaluated would make MAT scores dramatically increase. 

To reach tliLs conclusion we exa^^^ several peripheral issues con- 
cerning norm-referenced analyses and tn^TOie- third standard deviation 
criterion of educational significance. We pointed out several technical 
flaws in the norm-referenced "t" t»:st, and showed that, as applied to the 
MA.T, the criterion of educational significance was not a constant propor- 
tion of exp-^cted growth. In this sense the criterion was not equally 
stringent at all grades. 

Examining- our data relative tc the >iAT norms, we noticed that in the 
fourth grade it was more difficult to reach criterion than in the third 
and fifth grades. We executed computer simulations which confirmed the 
trend. We conclude that there is some artifact in the published norms 
at this grade. 

Based on our curriculum analyses and site visits we found that the 
PIPs did induce projects which were adequate copies of what v;as packaged; 
hov;ever what was packaged was not sufficient to implement the same curricula 
across sites. 

Our analysis of Lhc correspondence between the MAT and the curriculum 
materials wiiich were both listed in tlie PIP and used in the proiects 
provided evidence that, except at the lower grades, MAT items were not 
sensitvc to sucii materials. 

FLnalLy, in inir Least squares analysis, we found that teacher 
ros[)v)ns i. vencs s wns nivU'e ortoi^i associated with gains In test scores than 
v;a s go o d i.in. p 1 eni c n t n Lion of PIP specifications. 
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7 . 2 Met:liodo louLcal Reconunendat ions 



The preceeding sections have shown the utility of approaching 
evaluation through the use oC a principle description: the evaluator 
must display a connection beti^een the outcomes of interest and the 
treatment. This has implications for: 

1) The way USOE (or others) should decide that a project is 
effective, successful or exemplary. That is, the con- 
nection between the data offered as evidence of effec»-ive- 
ness and the content and procedures of the project should 
be judged for its reasonableness. 

2) The way in which PIP-type packages an? created in tht. 
future. A project could be analyzed from its outcomes, 
backwards to the proximal events which could have caused 
them, (all the way back to the management strategies 
which promoted such events if desired). The information 
in the package mi<7.ht then be more likely to convey the 
c f fcctive e lements . 

3) The way evaluation? are conducted. That is, to evaluate 
tlie effects of a treatm-..at appropriately would require a 
description L the treatment at the level of discourse 
relevant to Ihe efiiects examined. 

Our application of this idea to the evaluation of Project Information 
Packages led us to examine one standardized test and the associated norm- 
L-eferenced analysis in aetail. It was concluded that there were probably 
defec -s in that tests ^s fourth grade norms. We also found through our 
sLmulatioiis tliat "equ ipercen c i le " growth" could be achieved by guessing 
alone. v;e developed evidence that, in this study at least, compensatory 
c>ducation teachers do not teach to che MAT, except perhaps at the first 
and second grade levels. The test items however were sensitive to the 
responsiveness of the teacher, where responsiveness was judged by trainee 
observers and directly coded tor regression analyses. Similar analyses 
;-;iiv;ht be fruituil in other evaluations which use achievement test scales. 

linscd on '->ur results, wc v;ou Id recommend not using the MAT standard 
scores as zhc principle measure of project success. We would also recom- 
mend th.at the consumer of standardized tests not be drawn into the belief 
that standard i;r:eJ achLevement tests are equivalent, even i L' tests like 
the Anclior Test Study claim to display "equivalence" for some tests. 

We roc'^nuiKMid that the LmpUcations of assuming that any test's 
cross-sect L.MVil norms are lon^;l tudi:;a I ly valid be seriously considered 
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before the noniis arc used our; of level. We also question whether there 
is really a single trait called "Reading Achievement," and we question 
whether we know how it grov;s from the first grade to the 'v/elfth. If 
there is no such trait or we do not know its laws ^f growLh, then our 
achievement scale is nugatory. 

It seems to us that the trend of these considerations is r, .-rbandon 
norm-referenced, standarized tests with their simple scales. Wha" 
needed are tests with items that are sensitive to tnose skills we '.j-^-. 
taught. Tlien it it were determined that tc.achers were actually teaci 1 
such skills such tests would fonxi a convenii ; *:ouadation for unifOi.MJ 
evaluating diverse projects. 
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LNTRODUCTION 

The purpose of this manual is to explain the tasks and responsibilities 
of field data collectors who will be conducting tlie Spring 1976 testing 
program as part of the evaluation of Project Information Packages (PIPs). 
Recognizing the variations between PiPs and projects, an effort has been 
made to provide detailed directions to assure uniform testing procedures 
across all PIP projects. 

Uniformity is important for two reasons. Fir.it, for purposes of 
standardization, it is impo'.tant that testing procedures and conditions 
approximate, as closely as possible, those described by the test authors 
and publishers. Second, in order to provide reliable results, it is 
important to administer the tests as consistently as possible to different 
groups of students both within and across projects. Thus, it is important 
that field data collectors involved in the testing program thoroughly under- 
stand their tasks and responsibilities prior to assuming them and that they 
adhere to the guidelines for performing those tasks throughout the testing 
period . 

The field data collection staff will consist of the SRI Test Supervisor, 
the local Site Assistant and, where necessary, additional local personnel 
to serve as Testers and Monitors. The SRI Test Supervisor will assume 
overall responsibility for the testing program. Where possible, the SRI 
lest Supervisor will assume the role of Tester with the local Site Assistant 
serving as Monitor. In projects with large nmnbers of students to be tested, 
the SRI Test Supervisor will hire, train, and supervise local personnel as 
Testers and Monitors. Car will be taken to select people who do not have 
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a vested interest in the project. Local personnel who cannot demonstrate 
performance skills required by SRI, during on-site training, will not be 
utilized as a Tester or Monitor. 

The local Site Assistant will be responsible for completing all 
necessary preparations for testing, will assist during testing, and will 
be responsible for returning all test materials to SRI following completion 
of testing. 

I'h is maiiua 1 is divided in to four sec t ions . 
Section I . - Tests and Test Sample 

Section 11 - Site Assistant's Guide to Testing Preparations 

Section 111 - Tester's and Monitor's Responsibilities 

Section IV - Site Assistant's Instructions for Returning 
Test Materials to SRI 




Section i 



TKSTS AND TEST SAMPLE 



The test battery tur each student will consist of three types of tests 
1) The Metropolitan Achievement Test (MAT) 



a) For first and second grade^-s the FACES Attitude 
Inventory , or 

b) For third through ninth graders thf^ Intellectual 
Achievement Responsibility Scale ( LAR) 

A PIP and site-specific student attitude questionnaire 

a) For first and second graders, one which uses the 
FACES format, or 

b) For third through ninth graders, one wi.ich uses 
the Cooper smi th format , 



Metropolitan Achievement Test 

The Metropolitan Achievement Tests (MATs) are a s-^ries of measures 
designed to tell how much pupils have learned in important content and 
skill areas of the school curriculum. 

There are six levels of the MAT. The levels that will be used in Apri 
for each grade in the PIP evaluation are as follows: 
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One 



of two affective tests 



Grade 1 
Grade 2 
llrade 3 
Grade 4 
Grade 5 
Grade 6 
Grade 7 
Grade 8 
Grades 9 



Primary I 
Primary II 
E lementary 
E lemen tary 
I ntennedi^^ Le 
Intermediate 
Advanced 
Advanced 
Advanced 



There are three 
form F wi 1 1 be used , 



forms of the test (F, G, and H) at each level. Only 
Each member of the Test Battery has several subtests. 



F ACES A lt itude Inventory 

The FACES Attitude Iiiveiaory is designed to gvither information about 
the student's general feeling toward himself, toward others, toward school, 
and learning/, in general. la response to themes pictorial ly presented in 
the test items and verbally described by the Tester, each student shows 
his feelings by marking one of three responses: a happy face, ^ so- so 
face [not happy, not sad), or a sad face. There arc- 14 items in the FACES 
Attitude Inventory. 

Intellectual Achievement Responsibility Scale (lAR ) 

The Intellectual Achievement; Responsibility Scale (lAR) is aimed at 
assessing a student Vs belief in reinforcement responsibility in academic 
achievement situations. 

The I^VR scale consists of 20 forced-choice items. Oral preb'entation 
will be made by the Tester to students in grades three through five. 
Students in grades six through nine will be administered the lAR in 
v/ritten form. 

• 'Indent: Attitude Que stionnaires 

The Student Attitude Quests*. .;nnaires are designed to assess student 
teelinq.'^ toward the PIT projects. The Student Attitude Questionnaires are 
PI P- spec i t' ic and, in some instances, site-specific. The FACES format will 
be given to stiidencs in grades one cuid tV7o, and the Coopersmith Self-Esteem 
■*\vent. .-.ry *.':i\] bo givt^r L;j .students in grades tliree and up. 

Oral pre seTi la ti on will be made by the Tester to students at all grade 
levels. The Student Attitude Questionnaires should be administered immedi.ate 
Loliowing the affeciive tests. 



Tt :st Sample 

The test sample in April will include on'.v chose students tested In 
the Fall for whom we havp valid test data. The IRIT sample will consist 
of middle cycle students tested in the Fall. 

No additional students will be tested even though they are in the 
program. 

Table 1 shows the- v-rade levels that will be tested at each project as 
well as the subject area.r and affective tests each grade level will receive, 
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Section II 

SITE ASSISTANT'S GUIDE TO TESTING PREPARATION 



An Qverviev 

As Site Assistant, you will be responsible for all testing preparations. 
You should: 

- Check Test Rosters for accuracy and fill in missing information. 

- Prepare Test List worksheets when necessary. 

- Prepare a Tester Log for each Test Roster or Test List worksheet. 

- Gel the testing schedule approved. 

- Identify the location of students to be tested and enter teacher 
names on Test Rosters or Test List worksheets. 

- Locate testing areas and furniture. 

- Prepare school map. Duplicate for Testers and Monitors. 

- Duplicate approved test schedule. Distribute to teachers concerned 
and principal. Reserve copies for Testers and Monitors. 

- Locate space for local training session, if necessary. 

- Receive test booklets and check supp ly -against local needs. 

- Label test booklets and group booklets with appropriate 
Test Rosters (or Test List worksheets) and the Tester Logs. 

Test Roster-^What is it 

The Test Roster is an alphabetical listing of students, by school and 
grade, who were tested In Fall 1975. The Test Roster is furnished by SRI. 
The Test Roster will serve as the primary source of information in grouping 
children for testing, labeling booklets, and keeping a record of tesf:s 
administered. 

The format of the Test Roster is described below. A sample Test 
Roster is provided in Exhibit I. 

PROJECT: Project number (assigned by SRI), .^ame of the PIP 

and its location (city) . 

School number (assigned by SRI), and name of the 
school in which students were located in Fall 1975. 
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TEST GROUP: 
GRADE : 

TEACHER NAME: 

PUPIL NAME: 

BIRTHDATE: 

ETH: 

SEX: 

TUTOR/TUTEE CODE: 



VALID FALL 
75 TESTS: 



ABSENT 207o 
OR MORE: 



TEST SITTING: 

SEE TESTER LOG: 
COMMENTS: 



Test group Ldenti f ier--a letter such as A, B, 
C, ecc. or Z (assigned by SRI). 

CJrade Level of students at the time of Fall 
107'j testin^^- 

Blank column to be filled in according to 
instructions provided in this manual. 

Names of studei\ts tested in Fall N75. 

Birthdates of students listed. 

Ethnicity of student^J listed. 

Sex of students listed. 

Identifies student as a '.utor (1) or tutee (2). 
This column and the information in it is 
applicable only to the HIT sites. 

Identif. ^s those students who have valid test 
data (YES) from Fall testing and those who have 
no valid test data (NO) from Fall testing. ONLY 
STUDENTS WITH VALID FALL TEST DATA ARE TO BE 
TESTED IN APRIL . 

Five digit number for each student listed. 
(First two digits represent the project number 
and will be the same for all students in the 
project. The last three numbers represent 
unique student ID numbers assigned in the Fall). 
This column is to be completed according to 
directions in this manual. 

These three columns will be blank and are to be 
completed by the Tester. 

To be completed by the Tester. 

To be completed by the Tester. 



Test Roster--How to use it 

Correcting and Completing the Test Roster 

When you receive the Test Roster, the^irst thing you should do is 
verify the accuracy of the student names, birthdates, and tutor/ tutee codes 
(if applicable). Make corrections on the Test Rosters by crossing out the 
incorrect information and entering the correct information above it. All 
blank spaces should be filled in with the appropriate information. 
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Exhibit 1 
Sample Test Roster 



DAU RUNi OVIb/76 



> 
I 

H 



SRI PIP EVALUATION 
S'lRlNG m TfST ROSTFR 



PROJECl! Ig HIT S'^^ rlim'^ijL' 



SCHOOL! I Si,W>'i'/!l£ 
TtST GROUPi Z 
GPAOEl 6 



TEACHER 
.NAHE 



CIRCLE IF 



PUPIL NAME 
L*5T NAHE FIRST 


. E 
BIRTH T 
DMt H 


S TUTOR- 

E TUTtF 
X CODE 


VALID 
FALL15 
TESTS 


ABSENT 
20 X 

OR hORE 


TEST SEE 
SITTING TESTER 
•1 .2 .3 . LOG. 


1 m M 


06/l<)/6J a 


N 


1 


rEs 


32500 






()6/0S/6J B 




? 




32512 






; 1/07/1.2 fl 


M 


? 


YES 


32513 






f'./fli/M b 


F 


2 


TES 


325U 




mm to^H^T 


'ifl/JI/hil B 


H 


1 


YES 


32501 






l2/oyb3 B 


H 


1 


YES 


32502 






06/i;/61 R 


F 


2 


YES 


32515 






06/0S/61 B 


H 


2 


YES 


32516 





cohhents 



'er|c 
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You should then contact the project teachers and ask them to idCMitify 
those students who have been absent 207. or more during their PIP insLructioa al 
period . Do NOT ask the school secretary or regular classroom teacher for 
this information, A 207o absentee rate is approximately one day a week or 
a total of 35 days during the year ( 10 days absence during the cycle for 
IRIT project students). You should"circle the five-digit number on the 
Test Roster corresponding to the student's name for any student who has 
baen absent 207o or more in the column titled such. 

Filling in the Teacher Names will be explained in the Test Schedule 
section (page 14) of this manual. 

Identifying Test Groups 

The grouping of students into test groups has already been set up 
according to guidelines established by SRI. Tee test groups are the same 
as the test groups that were established in the Fall. The guidelines used 
were: 

First graders - testing in groups of 10 or less 

Second graders - testing in groups of 15 or less 

All other grades - testing in groups of normal class size. 

If the TEST GROUP identifier on the Test Roster is: 

A through Y - The list represents a test group. It should not 
be* changed. There may be more than one grade in 
one test group. If so this will be indicated 
on the Test Roster by your SRI Test Supervisor. 

Z - Pupils are grouped by grade within school. 

If the TEST GROUP identifier on the Test Roster is a ''Z'\ then the list 

of student may be used as is, may be combined vith other test groups to form 

a larger test group, or may be divided into smrill^.-r test groups. Your SRI 

Test Supervisor will indicate how the list is to be handled. If any list 
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has to be divided into smaller test groups, see the section Making up 
Test Li st Worksheets from Test Rosters below. Wliere two small groups 
are combined to make one test group simply staple the two Test Rosters 
together and they represent the test group. 

Test groups are not to be changed without consulting the SRI Test 
Supervisor. 

M aking up Test List Worksheets from Test Rosters 

As stated previously, the Test Rosters will sewn, in most cases., as 
the only lists of test groups you will need. However, the SRI Test Supervisor 
may indicate that Test List worksheets must be prepared and will attach an 
explanation to the Test Rosters explaining how students listed on the 
Rosters are to be grouped. It is your job then to prepare the Test List 
worksheets (see Exhibit 2) according to instructions. 

You will finJ that Test List worksheets will be made up only if one 
or more of the Test Rosters do not represent a test group. In m .i.ing up 
the Test List worksheets students from the same class should be grouped 
together. The Test Roster will serve as the source document and should 
be kept and return ed to SRI with the test bookets after testing has been 
completed. (Neither the Test Rosters or Test List worksheets are to be 
discarded. Both will be returned.) 

The top portion of the Test List worksheet contains space for recording 
the project number, name, and city, school number, school name, and grade. 
All of this information is on the top of the Test Roster and should be 
transferred to the Test List worksheet exactly as it appears. You will 
note that you are not to transfer any test group identifier. The name of 
the Test Administrator and the date of testing will be filled in by the 
Tester on the day of testing. 
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KxnibLt I 
TEST LIST WORKSHEET 




OJECT NO. 
SCHOOL NO. 



PROJECT NAME 
SCHOOL NAME 



CITY 



TEST ADMINISTrvATOR 



PROJECT IN FORMAT I 
PACKAGES EVALUATI 
Spring 1976 



GRADE 



DATE 



Teacher Name 


I.D. No. 


NAME 

Last First 


ETHNICITY 


SEX 




CTINGb 


Se£ 
Test 


1 


2 


3 






1 . 


















I • 








































































0 . 


















7 


















0 • 






1 












Q 

y • 


















10. 


















11. 


















12. 


















13. 


















14. 















ERIC 



Ethnicity Codes: B 
C 
S 
0 



Black 
Caucasian 
Spanish Surname 
All other 



Sex Codes: 



M = Male 
F = Female 
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The remaining portion of the Test List worksheet contains space £or 
the teacher name, student ID number, name-, ethnicity and sex. Filling -Ln 
the teacher name, will be explained in the Tes t Schedule section. Th^" 
other informat ion should be L rans f erred t o the Test List works* te t f ' om 
the Test Roster exactly as it appears. 

Please note that only sLudents witi» valid Fall i97j tests will be 
tested in April. Any siudenl wirli a *'N0" appcarin- tr^l column on the 
Test Roster should not have his name transferred to the Tesi. List worksheet. 
When the Test List worksheets have been completed, please double check to 
see that every student with a "YES" in the VALID FALL 75 TESTS column has 
been entered on a Test List worksheet. 

The columns headed SITTINGS and SEE TESTER LOG will be completed by 
the Tester during testing. 

Ethnicity and sex will also be verified by the Tester during testing. 
Preparing the Tester Lqk 

-X-Xe.ster Log must be prepared for each test group (see Exhibit 4, 
page 32). The purpose of the Tester Log is explained in Section III. 

The top portion of the Tester Log provides for projufM number, project 
name, city, school number, school name, and grade. Thir information should 
be transferred from the Test Roster exactly as it appears. 

If you must prepare separate Test List worksheets from the Test 
Roster, then you will fill in the top portion of each Tester Log exactly as 
you did the Test List worksheet. 

When you ire through, you will have sets of either Test Rosters and 
Tester Logs or Test List worksheets and Tester Logs. Each set will be kept 
with its associated set of labeled test booklets. 

A-17 



The Test Schedule 

You will be sent a test sched^;'. " prepared by the SRI Test Supervisor. 
Getting Approval of the Tes *. S '^^^e 

After you have received the schedule, present it for review and 
approval to the principal and regular classroom teachers. Please familiar- 
ize yourself beforehand with the Test SittinKS--Time Requirements description 
below so that you may answer questions regarding the need for the time 
requirements scheduled and explain che need to start each test session as 
scheduled. At some sites the operation must function like clockwork to 
work in all tests and all groups, giving each the fuli allotted Lime. 

After the schedule is finalized, duplicate and present copies to all 
concerned. Reserve enough copies for distribution to the Testers and 
ionitors . 

Identifying Location of Students 

While finalizing the test schedule, it is important to find out where 
the students will be during the scheduled testing times so they can either 
be picked up for testing or so the Tester and Monitor will know where to 
go to test them. 

Filling in the Teacher Names on the Test Roster or Test List Worksheet 
On the Test Roster (or Test List worksheet) enter the name of the 
teacher who is supervising the students at the time testing is scheduled. 
This information Is necessary so that Testers will know where to go to 
pick up the students during testing. If all students in the tett group 
have the same teacher, the teacher* s name need only be entered once on the 
Test Roster or Test List worksheet. 
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PreparinK a Diagram of the School 

Please prepare a diagram of the schools in which testing occurs. The 
diagrams should show the testing area(s), the teachers' rooms where students ' 
to be tested can be found, bathrooms, and the principal's office. Duplicate 
enough copies for distribution to Testers and Monitors. The diagrams will 
facilitate keeping to the prepared test schedules. 

Test Sittings--time requirements 

Table 2, Test Sittings, is a listing of typical sittings by grade, 
tests administered and the actual time allotted to each of the tests. The 
table does not provide for time between tests, time for distributing 
booklets, time to allow students to settle down before testing begins or 
for v.Uecting the booklets when testing is over. The schedules you will 
receive from the SRI Test Supervisor will include approximately ten minutes 
b.itween test administrations. This means that students should be ready to 
come to the testing location ten minutes before the tests are actually to 
begin. 

Locate Testing Area and Furniture 

As the testing plan is being discussed, inquiries should be made regarding 
tefting space and furniture since this may directly affect the testing 
schedule itself. The Site Assistant should ask to see the testing area, 

determine the adequacy of furniture, and make certain that school personnel 
who generally use the room are informed of its use as a testing area. If 
the testing area is ordinarily occupied by a school staff member, group, 
etc. , a test schedule should be provided to them. 
Space for Local Training 

A one or two day training session will be conducted by the SRI Test 
Supervisor in projects where local personnel are hired to serve as Testers 
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Table 2 



Primary I - Reading Subtest 

"Sitting I - What to Do (Practice) 

Re ading 
Sitting 2 - Word Knowledge 

Faces 

Questionnaire 



TEST SITTINGS 
Grade 1 



10 
30 
15 
15 
5 



Primary I 
Sitting 1 



Reading and Math 



Wliat to Do 



)wledgel 



Sitting 2 - Word llnowledge 
Faces 

Questionnaire 
..^Sitting 3 - Math Concepts 

Math Computation 



10 



if 
15 
5 
15 
15 



Grade 2 



Primary 


11 


- Reading Subtest 




Primary 


II 


- Reading and Math 




Sitting 


1 - 


What to Do 


10 


Sitting 


1 - 


What to Do 


10 






Reading 


30 






Word Knowledge 


18 


Sitting 


2 - 


Word Knowledge 


18 






Reading 


30 








15 


Sitting 


2 - 


Math Computation 


18 






Questionnaire 


5 






Math Concepts 


20 










"'Sitting 


3 - 


Problem Solving 


25 
















15 














Questionnaire 


5 



Grades 3*4 



Elementary - Reading Subtest 

Sitting i - What to Do 

Reading 
Sitting 2 - Word Knowledge 

lAR 

Questionnaire 



10 
25 
15 
15 
5 



Elementary - Muth Subtest 

Sitting 1 - What to Do 5 

Math Computation 35 

Math Concepts 25 

Sitting 2 - Problem Solving 30 

lAR 15 

Questionnaire 5 



Elementary - Reading and Math 

Sitting 1 - What to Do 10 

Word Knowledge 15 

Reading 25 

Sitting 2 - Math Computation 35 

Math Concepts 30 

^Sitting 3 - Problem Solving 30 

lAR 15 

Questionnaire 5 



* = day 2 
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Table 2 (continued) 



Grades 5-6 



Intermediate - Reading Subtest 

litting 1 - What to Do 5 

Word Knowledge 15 

Reading 25 

TAR 10-15 

Questionnaire 5 



Intermediate - Math Subtest 



Sitting 1 


- What to Do 


5 




Math Computation 


35 




Math Concepts 


25 


Sitting 2 


- Problem Solving 


25 




lAR 


10-15 




Questionnaire 


5 



Intermediate - Reading and Ma th 



Sitting 


1 


- What to Do 


f; 






Word Knowledge 


15 






Reading 


25 


Sitting 


2 


- Math Computation 


35 






Math Concepts 


25 


*Si tting 


3 


- Problem Solving 


25 






lAR 


10-15 






Questionnaire 


5 



dvanced - Reading Subtest 



itting 1 



Grades 7-9 



What to Do 
Word Knowledge 
Reading 
lAR 

Questionnaire 



5 
15 
25 
10 

5 



Advanced 
Sitting 1 

Sitting 2 



Math Subtest 



What to Do 5 

Math Computation 35 

Math Concepts 25 

Problem Solving 25 

iAR 10 

Questionnaire 5 



Advanced - Reading and Math 



Sitting 1 - Wliat to Do 5 

Word Knowledge 15 

Reading 25 

Sitting 2 - Math Computation 35 

Math Concepts 25 

^Sitting 3 - Problem Solving 25 

lAR 10 

Questionnaire 5 



= day 2 
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and Monitors. The training session will generally be conducted on the 
Thursday and/or Friday preceeding the week set aside for testing. The 
number of days set aside for training will be dependent on the number of 
grade levels being tested. The SRI Test Supervisor will inform the Site 
Assistant if a training session is to take place and will specify the 
date(s). 

The Site Assistant should arrange for a training location that will 
comfortably accomodate the participants. The Project Director should be 
consulted on a training location; After the training location has been 
arranged for, the Site Assistant should arrange to have all testing materials 
moved to the training location on the day training takes place and see that 
furniture is adequate. 
Receive Test Booklets 

Test materials required for each project are shipped from SRI to 
either the Site Assistant or to the Project Director. If the test materials 
are not sent directly to the Site Assistant, the SRI Test Supervisor will 
notify the Site Assistant as to where the materials were sent. 

Upon receipt of the test materials, the Site Assistant should open the 
test cartons and check the contents against the Packing Invoice (see Exhibit 
r, which will appear in one of the test cartons. Care should be exercised 
in opening the cartons so that they can be used for the return shipment. 

Upon examining the Packing Invoice the Site Assistant will notice 
that the top portion of the invoice, as well as columns. 1 and 2, will have 
been completed at SRI. The Site Assistant should count the number of tests 
received for each level that is recorded in column 1 and enter the number 
received in column 3. Columns 2 and 3 should agree. The Packing Invoice 
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Exhibit 3 
PACKING INVOICE 



PROJECT 



PROJECT NUMBER 



Entries are made on the Packing Invoice at SRI and at the local project. 

— Colui.:ns 1 & 2 are completed at SRI before the tests are shipped to 
the local project. 

Columns 3 through 6 are completed at the local project by the Site 
Assistant. 

— Column 7 is completed at SRI after the test? h. ve boen received back 
at SRI. 

This Packing Invoice must be returned to SRI with the test booklets. 

--Columns — 



Tests 
Name / Level 


Number 
Shipped 
From SRI 


Number 
Recv'd 
on Sit( 


Number 
Used 


Number 
Not 
Used 


Number 
Ro;turn- 
ed 


Number 
Rocv'd 
at SRI 


Comments 
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should then be checked against Table 1, page 6 of this manual, to insure 
that the correct tests have been received. Any inconsistencies should be 
brought to the immedia'ze attention of the SRI Test Supervisor. If thr- 
Test Supervisor cannot be reached, a collect call should be made to Nancy 
Craig, (415) 326-6200, ext. 2995. 

After testing has been completed, columns 4, 5, and 6 of the Packing 
Invoice should be completed by the Site Assistant. Column 4 ''Number Used" 
refers to the number of test booklets used by students, and column 5 "Number 
Not Used" refers to the number of blank booklets (this includes booklets 
which have labels on them but were not administered). Colunm 7 is reserved 
for SRI use and V7ill be completed at SRI as the returned tests are inven- 
toried and logged in. 
Labeling Test Bookieus 

After the test carton(s) have been inventoried, test booklets should 
be labeled and grouped for each day's testing. Each student's test booklets 
must be labeled before they are presented to him. 

The following procedures should be followed in the labeling process: 

1. Determine which group of students will be tested first and 
select the corresponding Test Roster (or List). 

2. Count: the number of students, listed on the Test Roister 
i'or List). Count out an equal number of the appropriate 
MAT and affective test booklets (lAR and FACES), and 
Student Attitude Questionnaires (FACES and Cooper smith) . 

3. For each, aame on the Test Roster (or Test List worksheet), 
select the corresponding printed peel-off student label 
generated at SRI. Place the label in the top ri^ht-hand 
corner of the* front cover of each test booklet . This label 
i s to remain on the test . The student's name is not to be 
recorded in any other space on the booklet. 

4. Review eacl. Lest Roster for corrections that were made and 
make ao:-*"pt*iat:e corrections on the student labels. 



Step 3 is to be repeated for each student on the Test Roster (or 

Test List worksheet). It is suggested that the MAT, affective test and 
» i 

attitude questionnaire be labeled at the same time for each student to 
reduce the possibility of error. The test booklets should be grouped and 
banded with the corresponding Test Roster (or Test List worksheet) in 
preparation for testing. At sites where there will be several Testers, 
the Site Assistant should group tests by school, test team, and test grou] 
or hour of test administration. 
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TESTER'S AND MONITOR ^S RK^'PONSIBILITIES 

The Night Before Each Day's Testln)^ 

The Tester must check the sets of test booklets to be used the next 
day against the individual Test Rosters (or Lists) to make sure that the 
right number of booklets are available for each grade level and that the 
names on the booklets match the names on the Rosters (or Lists), The yets 
of booklets should be put in order of use according to the next day's 
schedule. The list of required materials should be checked and materials 
assembled. See the list of materials below. Note that scratch, paper, for 
instance, is required for some math subtests but not allowed for others. 
At the Beginning of Each Day's Testing 

The Tester and Monitor should arrive at the school early, check in 
with the principal or school secretary as prescribed by school policy, 
and proceed to the testing location. They should have with them the 
following materials: 

- Testing schedule 

- Test booklets to be used that day 

- Test Rosters (or Lists) and Tester Logs for each set of booklets 

- Appropriate test administration booklets 

- Map of school room locations 

- Sharpened pencils for each student plus extras in case of 
broken leads 

- Scratch paper for math subtests where allowed 

- "Testing - Do Not Disturb" sign 

- Watch with a second hand 

- Two note pads (one for each Monitor and Tester to document incidents 
or disturbances that may affect test results). 
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Before the Students Arrive at the Testing Location 

The seating in the testing location should be surveyed to make sure 
there are enough chairs or desks. Seating of students should. be planned 
so that the Tester^can be clearly seen but borrowing of answers will be 
minimized. 

If it is necessary to gather and escort students to the testing i 

I 

room, the Monitor should review the Test Roster (or Test List) for location | 

I 

of students by teacher's name, and identify the teacher's location on the 
school map. If the students come from more than one classroom, the Tester 
will assist the Monitor in collecting and escorting students. 

Ten minutes before the test session is to begin, the students should 
be brought to the testing room. 
As Students Arrive and Before Testing Begins 

Students should be assigned seats as they enter, according to the 
seating plan. 

The Tester will introduce himself and the Monitor and will explain 

briefly the purpose of testing and the schedule of sittings for the 

particular group. The purpose of the testing can be explained: 

"As you know, you have been involved in a special program this 
year and we're interested in knowing just how much it's helped 
each of you. One way to find out is by testing and that's 
the reason you're here today. The tests you'll take are very 
important and I know that each of you will do your very best." 

If students know what is expected of them, they will be more able and 
willing to do their best. SRI will then, in turn, be able to obtain 
accurate estimates of pupil 'achievement. 

After introductory remarks, the Monitor will distribute the test 

A-27 



EKLC 



KJ O 



booklets and pencils by calling out the name of each student to ensure 
that each student receives the proper test booklet. As the student receives 
his test booklet, the Tester will verify, on the Test Roster ^^r Test List) 
the ethnicity and sex of the student noted, or fill in this information if 
it is missing. The Tester will put a check in the appropriate sitting 
column if the student is present. If the.^student is absent, the Tester 
will put a check in the "See Tester Log" column. The Monitor will then 
band together any undistributed booklets with the Test Roster (or Test 
List) and Tester Log. These will be set aside and testing may begin. 
Testing Procedures 

The following procedures must be followed to maintain an effective 
testing environment and provide uniformity of procedures among test groups. 

Maintaining Control 

The Tester should assume control of the group from the beginning and, 
at the same time, make every effort to maintain the confidence level of 
the students. The students will be reminded at the beginning of each test 
that they are not expected Co get every item right but that they should 
do the best they can. 

Once the test has started, all remarks should be grouped directed 

such as: 

"Let's all do our own work." 

"Let's all work quietly." 

"Let's do the best we can." 

"We're all working very well, etc." 

Remarks should always be in the form of a directive, never in the form 
of a quest ioa . 

A-28 



Practice Items 

Practice items are provided .on all tests, except the lAR and Student 
Attitude Questionnaire, to insure that the students know how to mark their - 
answers. As the students do the practice items, the Tester and Monitor 
should check to see that marks (X or blackened oval under, or next to, 
test items) are discernible, that they appear in the space provided, that 
answer selection is being completed quickly, and that only one selection 
is made for each practice item. 

If clarification is necessary, the Tester may demonstrate on the 
chalkboard how the students should mark their answers and may repeat the 
practice itera questions, 

Reading Test Directions 

Once the test has been started, the Tester must read the test directions 
exactly as they appear in the Examiner's co py. The Tester should never 
elaborate on the directions or provide his own interpretation. Neither 
the Texter nor the Monitor should provide a clue as to a correct or 
incorrect response in any manner (e.g., tor.5 of voice, facial expression, 
etc.)- If the Tester judges that most students in the test group did not 
hear or understand an item, the item may be repeated. 

Pacing 

On timed tests, students work at their own pace. If all students 
finish before the allowed time has expired, the test may be terminated. 
However, the full amout of time allocated for the sitting must be provided 
any student, who wishes to use the remaining time to work on the test. 
Students who finish early should be encouraged to remai quiet so tha^ 
those still working can complete the test undisturbed. 
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On liiicimed tests, the Tester should move the students along at a pace 
rapid enough to maintain their attention over the duration of the testing 
period ^i,e., allowing just enough time for the students to mark their 
answers, but not enough time to look ahead or back to previous answers). 

Monitoring 

The Monitor oversees the student testing activity to make sure that 
test results accurately reflect the capability of the individual student 
to respond. (The Tester will also assume the role of Monitor when not 
reading instructions to the students.) 

Monitoring is a very important part of testing. Possible problem 
situations and suggested reactions to those situations will be detailed 
below but, the ^^eneral requirements for good monitoring can be summed up 
as follows: 

• Be alert. 

• Keep moving within the testing room. 

• Do not help students with answers, but know ahead of time 
the page they are to be working on. 

• Know ahead of time the common testing problems that can 
occur. 

• Be willing to act immediately to remedy a problem situation, 

• Ise your pad and pencil to document those problems. 

Always be alert. This is the key to effective monitoring. Your eyes 
i-hould always be on the move, watching for problem situations. In addition 
to your eyes, you too should be constantly on the move within your designated 
area. (If the Tester is also monitoring, he should be responsible for 
half the room and the .Monitor the other half. This should be determined 
before testing be£;ins.) Never stand behind or beside a student and watch 
him work. You may pause a moment to check for problems, but move on 
quickly. 
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Never help students with answers but know aheac" of time what he's 
supposed to do. Listen carefully to the Toster so that you will know what 
pages tho stude.u must complete. II a student appears to have finished 
his test, always check his booklet to make sure all pages have been completed. 

Study the list of possible problem situations and prepare yourself 
to respond in the most positive and appropriate manner. Never single out 
students for praise or to express displeasure, but respond when necessary 
to remedy any situation which will negatively affect test results. Always 
minimize contact with students who v.-ant excessive attention. 

Always document on your note pad any situations which require your 
attention. This documentation will be needed to explain any test invali- 
dations which Will be recorded on the Tester Log following completion of 
the test sitting. Typical situations which might require test invalidation 
If not corrected are: a student borrowing answers from his neighbor; a 
student marking responses in an incorrect manner (marking multiple answers, 
marking the answers outsid.- the designated area); a student working on the 
wrong subtest; a student not finishing a subtest, thinking he's through but 
having more pages to complete; or exceptional -lat^sroom disturbances. 

There is no way to anticipate all the problems that might arise during 
a testing situation. However, there are certain jjuidelines which can make 
monitoring easier and more effective. Following aie some possible problem 
situations and suggested responses: 



1) Several students seem confused when the Tester is reading 
instructions. 

Monitor: Get the Tester's attention and quietly indicate 
the need to repeat Instructions. 

2) During an untimed test, students seem either restless or 
too rushed. 
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Monitor: Get the Tester's attention and quietly indicate 
the need to "go faster" or "slow down a little." 

Student is working on the wrong subtest or on the wrong 
p^'^.e, or turns more than one page when asked to "tuin 
the page." 

Monitor: Turn to the correct page in the booklet and 
place it in front of the student- 
Student is observed borrowing answers from his neighbor. 

Monitor: Act iminediately . Lightly place a hand on the 
student's shoulder and turn him back to his test- The 
Tester should direct a statement to the entire class 
"Let's all do our own work." If the student persists, 
quietly lead him by the hand to another seat, if available. 
(Do not, however, place the student outside the test 
group, i.e., in the corner of the nom.) If moving- 
the student is impossible and if borrowing persists, the 
test must be invalidated- Document the behavior and the 
student's name on your note pad- Arrange for different 
seating for the student on the next subtest or during 
the next sitting. 

A student is not marking responses in a correct manner; 
i.e., marking multiple responses to one question. 

Monitor: Move your hand across the range of choices and 
say "you have several choices but select only one answer 
for each question." (If the student persists, the test 
must be invalidated. Note the behavior and student's 
name on your note pad.) 

i.e., marking answers in the wrong place. 

Monitor: Move your hand across the range of choices and 
make an appropriate comment such as "Be sure you fill in 
the nnswer where it belongs," "Fill in the oval next to 
(or under) the answer you've selected," or "Put the X on 
the one choice you've selected," etc- 

i.e., marking only one answer when there are several 
questions connected with a story and there is a choice 
of answers for each. 

Monitor: Draw vour hand acrosn the range of choices for 
the c/uestion.s Left unanswered and quietly say "Be sure 
you answer all ihe questions to each story, if you can," 
or "No* u tb.ore is more than one question to each story." 
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A Student appears to be finished with the test but is 
in ract, not finished. ' 

Moni-tor- Wlienever a student appears to have finished a 
subtest, alvai^s check the subtest to make lure a pfg es 

uZ unT-T^.'""''' '''' unfinished. tuJn'to 

/ the unfinished section and draw your hand acros^ the 

or midd e^ nf '° P^^"^' ^° ^^e botLm 

(or middle) of page where it says STOP." 

.i™e'orM.'^L: iszVsTti:: Sn"°t"^°" " 

at the tl„a of his return ^ administered 

If students cannot be returned to the restinE area take 
t'TuTaZ "!ss"s';:r"°" ^r-'^' - 'hT^rutpal.s 

the students win ot the circumstances to ensure 

- tLir";:;uu^i^?:3:r^Tj::ie;"^ ^^^^ 

i.e.. a student asks if he can go the the bathroom. 
yoTc""wa?^:"^^ '''' ""^'^^ ^'^ -re 

i.e.. students finish cheir tests and become restless. 

ri^-^he"°f::nt":o::r""^:^ t-^ ^He"b"\r^ 

.uietly suggest the student .'dr^rs^L^S • n^^^^X tT.,, 

NOTE: A student should be removed from the testing 
group only if he becomes ill or his behavior 

II "° ^.T^'^^" ^^^^ ^'^ i= disturbing the 
rest of the group. 

Students seek attention from the Monitor. 

i.e., student asks if his answer is correct. 
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Monitor: Quietly say "Just do the bes. you can/' 

i.e., the stud..nt smiles at the Monitor each time the 
Monitor passes by. 

Monitor: Mav s.iU in return and .ove on ^ the student. 
Try to make an effort to minimize contact ^ith 
during the re.na;..vier of the sitting. 

Tl.is is not an exhaustive list of possible occurrances. but most 
incidents fall into the general categories described above. Responses . 
suggest appropriate monitoring behaviors. 
rnllecting , Test Booklets. 

,£,er an students have Einlshed their tests or the Tester Indicates 
the session is over, the Monitor <.nd Tester, it the .roop i, large, will 
collect the test hooUets (within their predesi.nated areas, hy »oving np 
one aisle and down the other until a te.t hoo.let (and pencil, if the 
Sitting ha. been completed, has heen collected Cro™ each student. 

U a second test booUet I, to be di.trihuted during the sa™e sitting, 
. the students „ay stand and stretcn for a few „o„ents before proceeding. 
*gai„, the Monitor should call out the na.es of the students to ensure 
that oath student gets the correct booklet. After the last test has been 
cpleted. the booklets are to be collected as described above. 

Students should be escorted to their cl.ssroo„s and re„inded to pass 
,„letly through the halls if other classes are in session. 

fUU;,,^ouj.IheJ^^^ 

The Tester Log n,ust now be filled out tor the sitting just completed. 
There should be a Tester Log for each test group, just as there is a Test 
Roster (or Test List Worksheet, for each. The top portion of the Tester 
Log Will have been filled out by the Site Assistant. The information will 
correspond to that on the Test Roster (or Test List worksheet,. 
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•■H.U„„ of Cro„p Te.. Co„d.Uo„s,.. "...v^u,,,,,, ,„,^^,^^ ^ „ 

..I„vaUda„o„ Of .„.w...„„ sc„„„„. s,,.,.,,.... ,,,, , 
^oUo«„, .He c„„pl.,.,„ or ^^^^ 
=ho„U .uc„,3 „„,^ _^ ^^^^^ 

a -u.e„. o. S.O.P of ,„^„^^ ^^^^^ 

Log, 

RaUn^^ Gr oup Test Cor ,d_i^^_nc 

The T,,.e. ..o„M „te H,e ^ ^^^^^^ 

by . c.,c. ,V, « „ 3c.no Of ,.c.n«.t e. poo.. P.«„. „. ^ 

-sf.„ed f„ 3.001. <„,i„.e c„„fo„ of ..e 

u,.hu„., ^.nuusy. ootside .,s.,.eu„„s. „ope.a»o„ of ,„„e„.. 



SO on 



The subtests of all students wUh.. . ,.oup ca. be invalidated, but 

-OiU. ir c.e.ti., conditions were so poor tHat ..e entire .roup was penaU.ed 

e.a.p.e. i. . are occ.rs during . .... .^st .nd s.HooI i« interrupted 

or dismissed, the Tester should record the u.e of interruption on the 

Tester Log and have the students close .heir booklets I^nediately. The 

booklets should be collected and an effort m-.K- ^ 

n ettort m.uh, to reschedule the remaining 

time allotted. if this is impossible then th. c 

H siDie, then the subtest must be invalidated 

for all membt. j of the grot p. 

The sa^e p.„ce.„.e. sHo.a. .e foUove. f„ the case of a„ l„.e„op„o„ of 
a. .nu„e<, .es. .ha. .e,of„. .he ..o.eo.s .o Uave .he cla.s.oo.. f„ .„f, 
case, however, .he f.e„ hel„s a.„f„fs.e.e. shoold he oo.ed o„ .he tes.er 
l^^, and an eff„„ .,de .o reschedule .he regaining ti^. 
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Exhibit 4 



TESTER LOG 




PROJECT INFORMATION! 
PACKAGES EVALUATION 
Spring 197^> 



PROJECT NO. 
SCHOOL NO. . 



PROJECT NAME 
SCHOOL NAKIi: 



CITY 



GRADE 



, n^ aive a general rating or ti- e„btest£ within 

-rri^rf^r^^cTT^ing. 



-^^[^^t Good Fair Poor 

I I L^. ^ ' ' 

I . 1_ 



Comments 



it tesciug ■ u„,,iH he made in tne uuim..^"-- - 

idation notations should invalidate a subtest wxU 

^* ^mTdTbTS^TI^^y^^^;''^'^^ conditions are: 

. or,,,;,! Invalidation coaeb auu ^..hrpst 
test manualo mv throughout mo 
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„ invalidation codes anu 

Student refuses to respond throughout most 
Student borrows answers consistently 
Student marks multiple answers consistently 
Scudent hecon.s iU during the subtest 
Student was absent 
Student worked the wrong subtest 
Suudent is in special education 

■ ,.fv in the consents section 

Q . Other - specitN m 



1 -- 

2 - - 

4 - 
D - 
6 - 
7 



, other - s„eci£y in the co^ents ,„v.,Ueated. Record 

„ be used onll i£ a student's ^""""^^^ ,ode that explains 

tbe reason for invalidating Che 

Reason for Cotnments 



Reason for 
Invalidation 




1 


2 


3 


4 


5 


6 


7 


8 


9 
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In either case, complete documentation should be made in the 
Comments section of the Tester Log. If space is insufficient, additional 
pages should be atLached. 

Invalidation of Individual Student Subtests 

Individual student subtests may be invalidaced by the Tester if one or 

more of the nine specified conditions exist. The specified conditions and 

associated code numbers are as follows: 

Code 1 Student refuses to respond throughout most of the test. 

Code 2 Student borrows answers consistently. 

Code 3 Student mar.is multiple answers consistently. 

Code 4 Student becomes ill during subtest. 

Code 5 Student was absent. 

Code 6 Student worked wrong subtest. 

Code 7 Student is in special education. 

Code 8 Student has severe mental/physical handicap. 

Code 9 Other - specify in Comments section. 

Codes 1, 2, and 3 should be considered only after the Tester or Monitor 
has tried several times, with no success, to get the student to respond, to 
stop borrowing answers, or to stop marking- multiple answers to a single 
question. Code 6 should be considered only if the student was not caught 
in time to start on the correct subtest. It should not be necessary to use 
codes 7 or 8 because those students should have been screened out of the 
test sample already. They are there to be used only in case of a screening 
error. Code 9 allows the Tester to invalidate a subtest for unforseen 
reasons not covered by the other codes. 

Code 9 should be used in case a student's test in invalidated because 
of a disturbance. Do not use Code 1 for such cases. Code 1, refusing to 
respond, is not the same as causing a disturbance. 
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If a student must leave the sitting for a doctor's appointment, for 
instance, then Code 5 should be used for any subtests he missed. Do not 
use Code 9. 

If a student is late to class and the test administration has begun, 
the student may be allowed to begin the test, aided by the Tester or Monitor 
to find the proper starting place. The word LATE and the name of the 
subtest being taken at that time must be written on the front cover of the 
student *s test booklet. The incident must be documented on a note pad also. 

There are two kinds of tests--the timed test which is not orally 
administered, and the untimed test which is_ orally administered. For the 
timed tests, after the booklets have been collected, the LATE booklet should 
be checked to see if the subtest which was started late was completed. If 
it was not completed, the subtest must be invalidated and coded 9. If it 
was completed, the LATE notation remains but no invalidation code should 
be entered. 

For the untimed test, there is no way a late start can produce valid 
test results. On the untimed, orally administered test, the student may be 
allowed to begin late simply to keep him occupied during the test period, 
but the test must always be invalidated. 

Any time a student's subtest is invalidated the student should be 
allowed to remain in the test location until the testing is completed, 
unless he is disrupting others around him. 

Steps to follow when invalidating subtests: 

1) You first observe a disturbance or unusual behavior which 
may affect test results* 

2) You attempt to correct the situation. 

3) If the situation cannot be corrected, you describe the 
occurrance briefly on your note pad, check the student's 
booklet cover and note his name also. 
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4) The decision to invalidate is made by the Tester after the 
test sitting is over. The Tester and Monitor discuss the 
occurrance and review their notes before the decision is made, 

5) The invalidation must then be completely documented. 

a) The name(s) of the subtest(s) invalidated should 
be entered on the front cover of the student's 
test booklet. 

b) The proper entries should be made on the Tester 
Log according to instructions below, 

c) A check should be put in the column "See Tester 
Log" on the Test Roster (or Test List worksheet). 
This will alert coders at SRI that there is an 
invalidation or unusual circumstance regarding that 
student *ssubtest(s). 

Whenever a student's subtest is invalidated, the following information 

must be entered in the boxes provided on the Tester Log. 

1) Student's name. 

2) Student's three-digit ID number, 

3) Name or names of subtest(s) invalidated during the 
particular sitting* 

4) Reason for invalidation checked (Codes 1-9). 

5) Reason for invalidation provided, briefly, in the 
Comraents section for any invalications other than 
for absence (Code 5)« 

A student's name may appear on the Tester Log more than once — up to 
as many times as there are sittings for his test group. 

The name of each subtest invalidated during a sitting must h^ entered. 
You should not note "sitting 1" for instance, rather you should note the 
names of the actual subtests invalidated during sitting 1. You should not 
note just "math" for instance, but should note which math subtests were 
invalidated, i.e., "Math Concepts," "Math Comprehension," or "Problem 
Solving." You should also specify whether "Word Knowledge" or "Reading" 
has been invalidated, or enter both names, if both were invalidated. 



SRI must always know the reason a subtest has been invalidated or has 
been left blank. A brief explanation must be entered in the Comments section 
each time an invalidation occurs, except for absence. 
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Ln all cases wher*- Test List worksheets have been prepared iron; ilu- 
Test Roster, please transfer the followiii>r. information from the worksheets 
to the Roster after all sittings have been administered to the y,vou\^ 
appearing on the Test List worksheet. Transfer all check m arks in the 
columns headed "Sittings," and "See Tester Log." All Information in the 
Comments sections should al-o be transferred to the Test Roster from the 
Test List worksheets. Remember--do not discard the Test List worksheets. 
They are to be banded together with the completed test booklets and the 
Tester Log. 
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Section IV 

SITK ASSISTANT'S INSTRUCTIONS FOR RETURNING TEST MATERIALS TO SRI 



Before shipping tests back to SRI, the Site Assistant must check the 
booklets against Test Rosters (or Test Lists) to make sure that all used 
test booklets have been accounted for, Do not change marks made by children , 
Packing and Shipping Booklets 

When testing has been completed for a test group, the test booklets 
and the appropriate Test Roster (or Test List) and Tester Log should be 
grouped together, rubber banded, and placed in the test carton. Place as 
many groups' tests in a carton as will fit. Tests from the same group 
should not be split into separate test cartons. 

All unused test booklets, both labeled and unlabeled, should be 
returned to SRI. The number of unused booklets should be noted in column 
5 of the Packing Invoice. Column 4 and 6 of the Packing Invoice should 
also be completed by the Site Assistant, 

Wien cartons have been packed for return, they should contain: 

1. Unlabeled test booklets. 

2. Labeled but unused test booklets. 

3. Test booklets used by the students. These booklets should 
be grouped with their Test Rosters (or Test List worksheets) 
and Tester Logs. 

4. All other miscellaneous test materials that were originally 
included in the carton(s) EXCEPT THE PRECUT SEALING TAPE 
AND RETURN ADDRESS LABEL(S). 

5. The Packing Invoice should be placed on top of the contents 
of one of the cartons. In those sites where Test List work- 
sheets were prepared from Test Ros::ers, the original Test 
Rosctrs .should be packed with the Packing Invoice. 

The te.st materials should be shipped to SRI via United Parcel Service . 

UPS will not send materials COD, therefore Site Assistants should bo 
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prepared to pay for the shipping costs and then include the cost on 
expense invoices to be reimbursed. The test materials should be shipped 
as soon as possible after testing has been completed. The Site Assistant 
should obtain a shipping number and estimated time of departure. The Site 
Assistant should then call Ben Samson collect at (415) 326-6200, ext. 3118, 

and report the shipping number, number of cartons being shipped, and estimated 

time of departure. 
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Appendix B 

CONVERTING STANDARD SCORES TO PERCENTILE RANKS 
AND DETERMINING THE EXPECTED SPRING SCORE 
FOR THE NORM-REFERENCED ANALYSIS 
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Appendix B 

CONVERTING STANDARD SCORES TO PERCENTILE RANKS 
AND DETERMINING THE EXELECTED SPRING SCORE 
FOR THE NOlU'l-REFERENCED ANALYSIS 



Percentile ranks are obtained from standard scores by using Table 3 
in the relevant MT Teacher's Handbook . Table 3 has beginning-of -year 
norr^s for fall testing and end-of-year norms for spring testing at each 
grade level for which the tests are appropriate. If a particular stan- 
dard score does not appear .in the table, the handbook instructs the 
teacher to use the percentile rank corresponding to the next higher stan- 
davd score. 

We determined this "rounding-up" procedure to be too insensitive for 
the norm-referenced analysis. An interpolation method was used to pro- 
vide more accurate conversion of PIP and site level mean standard scores 
to percentile ranks. The method, which takes into consideration the un- 
derlying normality of the standard scores, is as follows: 



(U 
(2) 

(3) 

(4) 



Find the two percentile ranks corresponding to next higher 
and next lower standard scores in the table. 

Look up 2 scores corresponding to the upper and lower per- 
centiles in a table of the standard normal cumulative 
distribution function. 

Perform a simple linear interpolation of the z scores to 
determine the z score corresponding to Che particular stan- 
dard score . 

Look up this 2 score in the normal distribution cable to 
determine the interpolated percentile rank. 



The expected spring standard score for a given fall standard score 
IS determined by a simple linear interpolation between the beginning-of- 
year and end-of-year standard scores. For example, if the fall mean 
standard score of 59.3 lies between standard scores of 59 and 62 in the 
beginnmg-of-year table and if these two standard scores correspond to 
standard score, of 64 and 66, respectively, in the end-of year table, 
then the expected spring standard score becomes 64 + (66 - 64) X fSQ l - 
59)/(62 - 59), or 64.2. ^ ^^^'^ 
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Appendix C 



CLARIFICATION OF PIP SPECIFICATIONS 
RESULTING FROM THF WASHINGTON CONFERENCE 



Catch-Up Group Meeting 
Washington, D.C. 
18 September 1975 

PROJECT CATCH-UP 
Smnmary of Revisions and Clarifications 



Public Relations 

• Daily contact between project teachers and regular 
teachers is vital. The teachers' lounge and che 
lunchroom are excellent places for such contact^ 
as is the playground. 

• One of the first faculty meetings of the year can be 
held in the Catch-Up lab. 

• Parental involvement is more difficult to bring about 
in some places than in others; less frequent involve- 
ment is expected if parents live a great distance 
from the school than if they live next door. Potluck 
dinners once or twice a year aro one way of getting 
parents involved. 

Staff 

• Teachers in Catch-Up should work four hours a day, not 
three, in order to v7ork with just two, three, or four 
students at a time. They are generally paid on an 
hourly basis. 

• Part-time staff is essential to the proj ect . 

• Aides do the complete job of instructors. 

• Teachers and aides should always maintain a positive, 
success-oriented approach in working with their 
students. Some suggestions are to provide a badly 
behaved student with an excuse ("You're too tired 
today. Why don't you come back tomorrow when you've 
had more rest.") and to encourage a poor reader to 
read simple books by saying he or she may someday 

be a father or mother and will want to read baby books 
to the children. 
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• Teachers and aides base their individual responsibility 
for the gains of 18 or 10 students on median, not mean, 
averages . 

Materials 

• The PIP listed eight years* accumulation of materials. 
All of this, of course^ need not be acquired by sites 
in the first year of operation. 

• Criterion-referenced tests may be used as teaching 
tools, this is usually done about once a week. These 
tests are a good means of keeping teachers on target 
in relation to individual students' needs. 

• The danger in using Catch-Up materials in the regular 
classroom is uhat they will lose their special status and 
may bore the students. 

• If materials are used that correlate with regular classroom 
materials, the child's confusion is reduced. 

• Random House math materials might improve Catch-Up 
math instruction. 

• Catch-Up is not the type of lab in which children 
move from one spot to another and in which materials 
and instructor are permanently stationed. Rather, it 
is a place where materials, instructors, and children 
all move about freely. 

• In scheduling, it works well for each teacher to have 
access to a particular machine or teaching tool on 
one day of the week. The teacher can decide whether 
to use the machine that day and, if not, can give 
another teacher permission to use it. 

Other 

• Parent aides are used primarily to help out in 
bilingual Jabs. Their use in regular Catch-Up labs 
is limited. 

• In some ways, the original Catch-Up design is geared 
toward bilingual Spanish-speaking children. This de- 
sign can be adapted to match new contexts; one site, 
for example, introduced .-naterials for black awareness. 
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Conquest Group Meeting 
Washington, D.C. 
18 September 1975 

PROJECT CONQUEST 
Summary of Revisions and Clarifications 



Public Relations 

Positive relations between project and nonproject staff are crucial 
in maintaining the necessary district support for the project. 

. The attitude of project staff should be that they are 
there to assist regular classroom teachers. Scheduling 
is to be worked out as much as possible to the con- 
venience of classroom teachers. In cases where a 
mutually agreeable schedule cannot be arranged, the child 
can be placed on a waiting list or in the control group. 

. Friday afternoons can occasionally be used to allow 

project teachers to observe regular classrooms and meet 
with teachers. Clinicians should take students' De- 
ficiency Checklists along to discuss with teachers. 

. Teacher.: report to principals at beginning and end of 
year and to parents three times a year (beginning, 
middle, and end). 

The importance of involving parents as well as nonproject staff 
was stressed. Various techniques have been found successful: 

. Invite parents, teachers, and administrators to chili 
suppers; "marathon meetings," where breakfast, lunch, 
and dinner are served; awards luncheon at the end of 
program. 

. Babysitting and transportation services should be pro- 
vided for parents. 

Staxf Issues 

The major issue was the role of the supervising clinician. 

• The main duties of the supervising clini-.ian are (1) 
to assist the project director with training and 
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administration, (2) to monitor and assist clinicians, 
and (3) to teach his/her own students. 

• The amount of time spent on any one of these duties de- 
pends primarily on the number of centers. At sites 
where the project is relatively small (e.g., fewer than 
10 centers), supervising clinicians will have their 
own students to teach and will spend one morning or 
one day per week monitoring and assisting clinicians- 

sites with a large number of centers (e.g., over 20), 
supervising clinicians may spend the major portion of 
the week observing clinicians and have no students 
assigned to them. 

• Calling the supervising clinician a Reading Coordinator 
or a Consultant and clarifying the role as one of assis- 
tant rather than supervisor will help avoid problems 
with staff resentment and union regulations. 



Trainin)?^ 



• Length of preservice training for the first year should 
be two weeks. For every year thereafter, new teachers 
should receive two weeks' training, and teachers con- 
tinuing in the project should receive at least one 
week. Preservice training should cover diagnosis 
only. Materials and remediation techniques should be 
taught just prior to remediation and during in-service 
sessions . 

• Each clinician is to administer, take, and interpr^^l. 
er?ch test during preservice training. 

• During training, clinicians should work through a 
sample case from diagnosis up through remediation. 



Instruction 



• Diagnosis 

- The following 12 diagnostic instruments are to bu 
used. The importance of administering the instru- 
ments in this order was stressed by the originating 
project director. Tests that do not appear on this 
list have been delet^id from the battery. 



C-6 



General Information Sheet 
Teacher Referral 

Health Screening--Audioineter & Titmus (to be ad- 
ministered by nurse) 

Slosson Intelligence Test (SIT)^ 

S lesson Oral Reading Test (SORT) 

California Ach, Test"*^ (reading rooms); Gates- 
MacGinitie*'^ (clinics ) 

Bond-Baiow-Hoyt (reading rooms); Stanford Diag- 
nostic Test (clinics) 

Spelling Inventory — Betts (reading rooms); Kctt- 
meyer (clinics) 

Reading Inventory--Inf ormal (reading rooms); Sub- 
jective (clinics) 

Programmed Reading Placement Test^^ 
Interest Inventory 
Deficiency Checklist, 

- It was agreed that it is the process of diagnosis 
that is important rather than the specific tests 
used. The idea is to get the information provided 

by these instruments while administering the smallest 
number of tests possible. 

Replicating sites have found certain other tests 
useful (e.g.^ the Wepman or Peabody for visual/ 
auditory discrimination; Fountain Valley for word 
recognition) . 

• Individualization 

- Remediation proceeds from the Deficiency Checklist. 
Teachers should find romprehens ion passage ap- 
propriate to each deficiency, then work backwards 
to vocabulary and phonics. 



Results from as much as two years previous may be used. 

t 

If a score is avaij.able from a test providing national norms on vocabu- 
lary and comprehens-^.on., this test is not needed. 
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Instruc -.1011 should be adapted to student's 
interests, as revealed by Interest Inventory. 

Individualization depends on detailed record keeping. 

• Instructional facilities 

Principals should be involved in designation and 
allocation of facilities before start-up of program. 

The reading room/clinic combination should be in 
one room; separate centers should be housed in 
separate rooms or portable facilities. 

• Instructional materials 

Regular district basal series is not to be used. 

Programmed reading series should be Sullivan, 
unless Sullivan is being used in the regular class- 
room. If this is the case, another programmed 
series covering comprehension may be substituted. 
It is important that one continuing series be 
used dai ly . 

To reinforce skills, move from Conquest to Dr. 
Spello. 

Conquest workbook is not to be used as a consumable. 

- When there is a suggested order in materials (e.g., 
the Phonovisual ) , follow the sequence described. 

- The guideline is 10 minutes on each activity, but 
this is not rigid. At the beginning cf each period, 
5-10 minutes should be spent establishing rapport 
with the children. 

Record Keeping 

• To achieve consistency, use one system for primary and 
one for intermediate. These syvStems are to be worked 
out by the teachers. 

• Clinicians should take three minutes at the end of each 
period to record what children have done. 

Reading Rooms Versus Reading Clinics 

• ■ Struct'j.re and format are the same, 

• Materials differ according t:o level, 
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KIT Group Meeting 
Washington, D.C. 
18 September 1975 



PROJECT HIGH INTENSITY TUTORING (HIT) 
Sunimary of Revisions and Clarifications 

Public Relations 

Good public relations lead to long-term survival of the project; 
therefore, 

• The project director must ensure that principals are 
consulted in matters that affect their schools as a 
basis for winning their long-term support. The 
principals should gradually gain a sense of owner- 
ship of the project. 

• Project teachers should ensure that their colleagues 
support the project. Support is more likely when 
classroom teachers are involved in the selection and 
scheduling of tutors and tutees and when project 
teachers maintain daily contact with the instructional 
staff. If project teachers are new to the school, 
winning respect requires extra effort. 

Tutors /Tutees 

• Tutors should mainly be eighth graders and tutees 
sixth graders. The age difference was found to be 
important at the originating site. 

• The maximum number of tutees to serve per period in 
HIT is 12. More tutoring pairs are" hard to monitor 
effectively and paperwork is excessive. Since the pro- 
gram is designed to benefit tutors as well as tutees, 
this still allows 24 students to be served in each 
period . 

• The tutor pool should be about twice as large as the 
number of tutees. Tutors usually come three times a 
veek; none come only once. They cc>rr:C: at different 
tiines so Whey do not miss the same clas:5 more tha-. 
cncc a week. Tutors arc never taken out: of reading 
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and math classes. Tutors make up homework for classes 
they miss. It becomes easier to recruit tutors if 
administrators adopt a policy that they may miss a 
class to come. 

• Tutees do not make up homework in classes missed; the 
cooperation of the classroom teacher is essential. 

Instruction 

• Each center should have only five half-hour tutoring 
sessions per day. Longer sessions should be avoided 
if possible, since it is difficult to maintain the 
intensity level characteristic of HIT for more than 
half an hour. 

• . Materials should be on student desks when they 

arrive for tutoring. Tutoring should begin right away 
without any distractions from the teacher, such as 
roll call. Absences can be noted from unused student 
folders. Materials should be placed where tutors can 
pick them up as needed without the teacher's or aides' 
help. Tutoring should be going on during the entire 
period with no time allocated to pep talks, discussions 
of discipline, or delay for passing out materials. 

• Record keeping and setting out folders and materials 
should occur dur.ing breaks between each session. 
Teachers and both aides should be walking through the 
room listening to tutoring and helping as necessary 
while tutoring is going on. Teachers and aides 
occasionally tutor to help students having special 
problems. Enough tutors should be recruited so that 
teachers do not perform this role, merely because 
there are not enough tutors. 

• No written answers or drill should be assigned in HIT 
reading centers. This would slow down oral reading 
practice and be tedious to students. 

• Avoid lengthy explanations that take time students 
could use in active skills practice. 

• Students may jump several pages in ^Sullivan if they 
gec over 947a correct for three days. If they pass a 
section test in the middle (or at the end) of a book, 
Lhey may jump to the next section (or bock). At the 

beginning of the yo.^ir Ihc.y may jump tlirough mai)y 
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books in a few days if they did poorly on the placement 
test but are able to do the work. 

When a tutee, such as a hyperactive child, does not 
respond well to programmed and drill materials, the 
teacher may work with him to diagnose his or her read- 
ing or math problems and assign other types of work. 
It helps to assign very patient and skilled tutors to 
work with such students. 

Teachers should not be concerned about tutees "peeking"^ 
at answers; they should be getting 90%-947o correct and 
can learn by looking up the others on the answer sheet. 

Never publicly correct students, especially tutors, 
for misbehavior during a session. Talk to tutors 
after class in a private room, alone. If they do not 
want to be tutors, replace them. Make tutoring de- 
sirable by treating tutors as paraprof essionals . 
Emphasize that they are teachers, too, but will not 
have permanent positions if they do not accpet the 
responsibility. In the beginning of the year, nominate 
them as "chosen" rather than asking for volunteers. 

When t-Jtees make an error, an "H" for "help" may be 
placed on the tally sheet instead of a zero, which has 
more negative connotations for students. 



Materials 



If older or younger students attend HIT, materials 
other than Sullivan may be needed for them. Materials 
should be selected that lend themselves to fast-paced 
tutoring. They should have: 

- Simple directions or a repetitive layout that 
makes directions unnecessary to repeau. 

- Answers easily found and read by tutors (not 
in tiny print in a dif f icult-to- f ind teacher's 
section) . 

- Unambiguous answers (no subjective questions with 
several possible answers for tutors to judge). 

Drill in math uses cnany different materials designed 
to teach basic facts. These may be coirimeixlal or 
teacher-made. Records on which sets of facts each 
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student has mastered are kept in the student folder. 
Teachers should devise sufficient drill materials 
. (e.g.^ six or seven ways of teaching multiplication 
tables, and exercises such as measuring the room to 
learn the metric system) to keep drill from being 
unnecessarily monotonous. Points are given for drill 
partly so that the error rate in drill is controlled 
at 907.-94% correct. 

• Drill materials used in reading can include word lists 
derived from lists in' Conquests in Reading , Why Johnny 
Can't Read , or other sources. Drill in reading is 
aimed at fluent word recognition. Spelling or writing 
words letter by letter interrupts this fluency and 
should not be done. Students should work with a set 
of words until they can read 907o-947c, of them fluently 
without stopping to decode them. Variation can be 
added to drill by occasionally making a game or con- 
test out of drill words. 

• Occasionally for variety in reading, auxiliary materials 
such as plays may be used in place of programmed readers. 

Rewards 

• Attractive certificates printed locally can be given to 
tutors at the end of the year, as well as awards for 
the highest achieving boy and girl tutee and tutor. 
Tutors can also be given holiday presents, ouch as 
small wallets or bracelets. Only tutors who have come 
at least a minimum number of times should come on 
field trips. It is Helpful to consult with tutors re- 
garding the types of rewards they would like (e.g., 
\^ere to go on field trips). 

• One point is awarded for each drill item and one 
point for each problem in math and sentence in read- 
ing. If too many points are earned for the reward 
budget, the "price" of items (in points) may be 
changed . 

• Tutee rewords may include fruit, model airplanes, 
cafeteria passes, or other items, but attractive cjindy, 
such p.r, chocolate bars^ should be among the optionr, 
offered students each time rewards are given. Teachers 
should decide how to spend money for rewards. 
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Tutors may receive 200 points for each session they 
attend. They use these points toward candy rewards 
This is optional.^ 



Other 

• New students may join the program mid-year if there 
is room. They begin at their level in Sullivan, 

• Students who complete all the programmed materials may 
drop out of the formal program mid-year^ but the 
teacher should continue to see these students fre- 
quently, invite them to visit;, and ensure that they 
feel they are receiving special attention even though 
they have finished with Sullivan. 

• Be wary of including special education (retarded) 
students in HIT if these students are not integrated 
into all classes. Many HIT students may be sensitive 
to being labeled slow learners. The project is not 
designed for special education students. 
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IRIT Group Meeting 
Washington^ D.C. 
18 September 1975 



PROJECT IRIT 
Summary of Revisions and Clarifications 

Ge neral 

• Good relations with the sending school principals and 
teachers and with district administrators are critical 
to project success. Maintaining good relations is an 
important part of the jobs of IRIT project directors 
and teachers. 

• IRIT should make an effort to help regular teachers 
improve their teaching of reading. Intern programs^ 
demonstrations ; and inservice sessions were de- 
scribed. 

Project Organization 

• Wuure practicable; drawing all 45 st-^dents in a cycle 
from one school is preferred--both by sending teachers 
and for logistics. 

• When all IRIT students from a single classroom attend 
the same cycle^ the regular class sir.e is significantly 
reduced. This has proved to be one of the major 
attractions of IRIT and should be tried in new sites 

if at all possible. 

Student Selection Process 

• Br !:orp. each cycle^ the entire te.am should meet with 
S'Snding i:ojvr:t-:i.-s to '::xplc:i-.. the project and start the 
selection process. This m.eeting is a critical part 
of establishing good relations. 

• Teachers are asked to nominate about 60 students. 
The IRIT team selects ^5. 

• No student should start more than two weeks late. 
Students dropping out after this time should not be 
replaced . 
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• The first cycle should start about two weeks after 
the beginning of school in the fall. This allows time 
for* the selection process as well as some training 
and room preparation. 

• The last cycle should end as near to the end of the 
regular sessions as possible to minimize disruptions 
to sending rooms at year's end. 

Instruction 

• Coordination of the three reading areas requires 
daily meetings of the team. Decoding can be used as 
the core for discussing each student. 

• Individualized reading- requires an exceptionally 
energetic, personable, and creative teacher. 

• IRIT teachers generally specialize in one area 
rather than rotating from area to area within a 
givtn ye.ar, but may change from year to year. 

• IRIT trains its own substitute teachers so that they 
will be familiar with IRIT procedures. 

Materials /Equipment 

• Basal readers should be integrated into the curriculum 
if the rfigular teachers so request. They should be 
incorporated into the IRIT approach, but not to the 
exclusion of other core IRIT materials. 

• Whether or not the basal reader is used in IR.IT, the 
regular teacher needs to know where to place each 
student after the cycle. IRIT teachers should give 

an appropriate placement test at the end of each cycle 
and advise the sending teacher 

• IRIT does not place excessive emphasis on the use of 
teaching machines, although they can be useful for 
practice and as motivators. 
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PTR Group Meeting 
Washington, D.C. 
18 September 1975, 



PROJECT PROGRAMED TUTORIAL READING (PTR) 
Summary of Revisions and Clarifications 

Seven topics were discussed by the participants in the PTR group 
at the replication conference. They are reported in their order of 
importance to the discussants, as perceived by the recorder. 

Teacher Participation in Selection of Students for PTR 

• Two objections were raised to including teachers 

heavily in the student selection process. First, it 
was noted that including teachers* subjective opinions 
about students as a basis for selection could (in 
Dallas, would) corrupt the local ev^iluation design. 
Second, it was felt that asking for teachers' judg- 
ments for selecting students was an inappropriate 
way to elicit their approval for the project. The 
way to select students, it was agreed, is to use test 
scores. Teachers who strongly disagreed with the selec- 
tions made this way could review test results and dis- 
cuss their opinions with the director or supervisor. 
If a strong case v/as made, arrangements could be made 
to retest the student in question. This was the pro- 
cedure used in Farmington when PTR was validated. 

• Recent refinements to the student selection procedure 
used in Farmington were discussed. They now use a 
combination of test scores and teacher rating (without: 
teacher review of the test results before the rating). 
This method was considered unfeasible for other loca- 
tions, as Farmington has a computer program to com- 
pile the raw data and a rather complicated formul:-i for 
incorporating teachei rating with test scores. Farm- 
ington now uses spring posttest scores in selecting 
students rather than administering a pretest in the 
fall. (They tutor the first four grades.) In selec- 
tion also, the previous yearns teacher ratings are 
used, not the rew teacher's ratings. 
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• In terms of selecting children, something should be 
said about the populations being served. Farmington 
is predominantly middle-class and rural. The Dallas 
population is low income, predominantly black, and 
inner-city. Canton has a similar population, except 
it is rural. Farmington reportedly had a number of 
students "top out" on the pretest (Murphy-Durrell), 
whereas in Canton a significant number "bottomed 
out." In Dallas, most were at the low end of the 
scale. 

Teacher Support for PTR 

• Discussants agreed that it was virtually impossible to 
create strong teacher support for PTR prior to its 
operation for two reasons: First, they cannot yet see 
the value of the project; second, teachers are in- 
herently threatened by the prospect of others, partic- 
ularly those not members of the guild, "teaching" 
their students in a subject as fundamental as reading. 

• The best that can be done is to explain PTR as fully 
as possible to teachers before the tutoring begins, 
especially pointing out that the tutors do not initiate 
teaching strategies but are told exactly what to do 

by the tutoring programs. In Farmington, teachers 
were brought into four orientation sessions before the 
project began, on a paid basis • This option may not 
be economically possible elsewhere. In any event, all 
three project directors agreed, by the end of the first 
year of operation the teachers oven^^helmingly supported 
the project because of the results it had achieved, both 
cognitively and affectively. The replicating sites 
felt that the only way to elicit teacher support was to 
demonstrate the project for a year. This meant man- 
dating it and trying to hold the line until teachers 
realized the positive aspects of the program. 

The Alphabet Skills Book 

• All three project directors have concluded that The 
Alphabet ?' <ills Book is indispensable and mu c be in- 
cluded in the PIP. When tutored students -lo not 
know the alphabet skills, the tutors invariably become 
frustrated and the time is wasted. Both replicating 
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sites obtained copies of The Alphabet Skills Book and found 
it absolutely mandatory for children without readiness skills. 

• The Farmington site has been unsuccessful in its effort 
to secure permission to ir»c.orpc rate The Alphabet Skills 
Book into its program. As a result, Farmington is de- 
veloping its own readiness skills book, which would 
represent a useful addition to the PIP. The Farmington 
PTR staff use this readine:;s skills book exclusively 
for the first 6-8 weeks at the first grade level to pre- 
pare children for the tutoring materials that correspond 
to basal readers. This also delays introduction of the 
tutoring kits long enough to minimize the possibility 
that tutees will move through the basal readers more 
rapidly than other students in their classrooms. 

Training; 

• The present training is not sufficient. There should 
be separate training materials for each publisher's 
tutoring kit. 

• A suggested sequence .if training steps is: preview^ 
explanation^ demonstration, practice. Each task or 
small group of related tasks should be separate instead 
of on the same tape sequence. Each step should be 
more fully explicated. There is not enough practice 
time allocated in present training as implemented in 
sites. Farmington had "outside training support (Uni- 
versity of Indiana) for the first 2-3 days. 

Parent Involveme nt 



• Discussion on the issue of getting parents involved re- 
vealed that the differences in socioeconomic levels 
between the originating anri the replicating sites were 
vast. The originating site found it much easier to in- 
volve parents, who were typically middie-class. In 
the depressed areas the replicating sites served, it 
was almost impossible to get parents involved. Some 
suggestions for drawing parents out Included: 

Paying them to attend meetings 

Inviting them to actual tutoring sessions 

- Holding parties, with refreshments, at which their 
children read 
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- Giving rewards to parents of high achieving tutaes 

- Coordinating PTR with the Reading Is Fun damental 
free book orogram 

- Taking native language ttipe/slide presentations to 
parents ^ homes . 

• An encouraging note was that, regardless of parent in- 
volvement during the year, there was strong support for 
PTR among parents, who reported that participating 
children read a great deal and held positive attitudes 
toward themselves c?.nd learning because of the program. 
In fact, had it not been for parent enthusiasm, the 
program would not have survived past the first year in 
Farmington. Test score gains were nil the first year 
of the program, according to Dallas Workman, However, 
the program proved strong in developing positive at- 
titudes among tutored children. 

Compatibility of PTR and Nonpublisher Basals 

• Dallas Workman reported that many Farmington children 
who participated in PTR have their basic reading pro- 
gram in basals not connected to PTR-associated pro- 
grams. Children in these basal programs show no dif- 
ference in achievement gains from children in the basal 
reading series that match the PTR program they use 
(i.e., the Ginn 100 is used in some classrooms, whereas 
the Ginn 36^ tutoring kits are used; some tutored 
children have their basic reading programs in texts not 
published by : I R kit publishers, as well)* 

Technical Assistance 

• The three project directors agreed they need to have a 
resource person whom thev can call when confronted with 
unprecedented project problems in both instruction and 
management. Replicating site directors were grateful 
for Dallas Workman *s suggestions on the phone. All 
three agreed to continue to support and be on call for 
one another and for new replicating sites. 

• Dallas Workman felt that, for future PTR replicating 
projects, two contacts were essential. 

- Someone at the federal level (monitoring agency). 
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Someone capable of giving technical advice based on 
experience (i.e.^ Farmington, Canton, Dallas). 

• Dallas Workman also felt that^ if the program was to 

succeed as a project, it* was essential that the projects 
have the latitude of "adapting" the PIP to local con- 
ditions . 
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R-3 Group Meeting 
Washington, D.C. 
18 September 1975 



PROJECT R-3 
Sunrmar'' of Rev ir> ions and Clarifications 

Gaming/S imulationg /Contracts 

• Summer workshops in gaming/simulations should be planned 
(for cadre teachers) to get a head start on adapting and 
integrating gaming/simp.lation into the curriculum. 

• Gaming/simulation activities should be incorporated into 
the reading, math, and 5'>cial studies curricula as often 
as possible, 

• Contracts are an integral part of R-3. Contracts can 
be purchased as well as developed by teachers. 

Instruction/Mater ials 

• Overuse of ore instructional method deadens the effec- 
tiveness of the approach, 

• Two planning periods a day are needed by all R~3 
teachers to plan the curriculum and coordinate asiong 
the teams and team members. 

• Project director autonomy is essential to Project R-3 
success. The project director must have budget con- 
trol to purchase materials needed by teachers. 

• Diagnostic-prescriptive materials should be used ex- 
tensively in reading and math, although activities 
should be changed to reduce or eliminate boredom, 

• Barbara Evans (Project Director, Lorain, Ohio) agreed 
to send lists of contract materials she found useful 
as well as reading materials used foi the diagnostic- 
prescriptive technique . 
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Public Relations 



• Home visitations must be conduct-d early in the school 
year. A great deal of effc i: is needed in obtaining 
parents' understanding of gaming/simulation and the 
R-3 philosophy. 

• Greater efforts will be made to help other teachers 
understand the R-3 projects; at the same time, R-3 
project teachers vill continue to plan together as a 
group. 
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Appendix D 

INDEPENDENT VARIABLES FOR THE REGRESSION EQUATIONS 
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INDEPENDENT VARIABLES FOR THE REGRESSION EQUATIONS 



The independent variables for computing ^the regression equation for 
each grade level for reading and math are indicated by X's on Tables D-1 
and D-2. For example, the regression equation for grade 1 reading in- 
cluded seven variables: (1) age, (2) sex, (j) minority status, (4) good- 
bad rating for Catch-Up teacher responsiveness, (5) well-poor rating for 
Catch-Up teacher implementation, (6) good-bad rating for PTR (Dallas) 
teacher responsiveness, and (7) well-poor rating for PTR (Dallas) teacher 
implementation. 



D-3 



Table D-l 

INDEPENDENT VARIABLES FOR THE READING lU-GRESSION EQUATIONS 





Grade 


Vnri.ablc 


1 


1 

(Can ton) 


2 


3 


A 


5 


6 


7 


8 


8 

(R-3) 


CouCinuous 






















Age 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Indicator 






















Sex 

/O = Female \ 
I I = Male / 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


Minor L ty 

/O = Cauca'sian \ 
\l = Black or Spanish/ 


X 




X 


X 


X 


X 


X 


X 


X 




Black 

/O = Caucasian or Spanish ^ 
\ I Black / 




















X 


Whi ce 

/O = Black or SpanishN 
\ I - CuTucas ian ' 




















X 


Wo 1 1- implemented PIP (sec- Section 6.2) 
/O = Not well implemented \ 
\ I ^ Well implemented ' 

Catch -Up 

Conquest 

HIT 

IRIT 

PTR 

R-3 


X 

* 

X 




X 


X 


X 
X 


X 
X 


X 
X 


X 






Responsive Teachers (see Section 6.2) 
/O - Not rf.'spons Lvo \ 
\ I = Responsive / 
Catch-Up 

Conquest y 

HIT * ^ 

IRIT 

PTR 

R-'3 


X 
X 


X 


X 


X 


X 
X 


X 
X 


|x 

X 


X 




X 


Combination of" uv> 1 1- Imp lemen ted PIP .-and respon- 
sive teachers (see Section 6.2) 
/O ^ Poorlv implemented or not responsive \ 
\l - Well Ltpplemented or responsive / 

Catch-Up 

Conquest 

HIT 

IRIT 

PTR 

R-3 






X 


X 
X 


X 




X 




X 





"liisuCf icicnc (Intn for Che PIP at this grade level. 
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Table D-2 

INDEPENDENT VARIABLES FOR THE MATH REGKESSION EQUATIONS 



V«ir i.ib les 



Cone inuous 

Age 
Indies Cor 

Sex 

\ L - Mnle / 

Minor ity 
/O - Cuicnsirin 



Block 

/O - Caucnsi.in ov Spnnisl- 
\ I = Black 

Whi re 

(0 = Black ur Spanish \ 



Grade 



1 

(Can ton) 



We 1 1- implemented PIP (^ee Section 



/O = \'ot well implemented 
Vl - Well implemi'nted 

Cntch-Cp 

Conquest 

HIT 

IRIT 

PTR 

R-3 



Kesponsivte tcncher.-^ (see Section 6.2) 

0 = Not responsive \ 

1 - Responsive I 
Critch-Up 
Conquest 

H Vl 
IRIT 
FTR 
R-3 

Combination of we 1 1- implemented PIP and respon- 
sive teachers (see Section 6.2) 



/O = Poorly implemented or not responsive 
\l = Well implemented or responsive 

Catch-i;p 

Conques t 

MIT 

r.RIT 

PTK 

R-3 



X X 



Insiifticient dni^ tor t.he ?IV :u tills ^rade level. 
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Appendix E 



MOST FREQUENTLY USED MATERIALS AND SKILLS EMPHASIZED IN CATCH-UP, 
CONQUEST, HIT AND IRIT PROJECTS, BY GRADE LEVEL 
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HOST nWW USED MATERIALS AND SKILLS EMPHASIZED IN CATCH-UP PROJECTS, BY GRADE LEVEL 



a. Grades 1 mil 2: Reading 



Catch-l| Haterial 



Published reading materiiil 

1. Instant Readers 

2. Library (free reading) 

3. Gateways to Reading Tre:isures 

4. Scholastic Individualized Reading 

5. Sounds ot Language Readers 

6. SRA Reading Laboratory; 
Ky Own Book 

7. SRA Reading Program 

8. Systems 80 

9. Random House Criterion Reading 

10. First Talking Alphabet 

U. Open House Series'-More Power 

12. Phonics i-,'e Ose (Old h Nev) 

13. Phonics We Use Learning Gaes r-'lt 

14. Specific Skills Stries 

15. Alpha Bingo (ga^f;) 

16. Dolch Basic fcca List 

17. Letter Recognition 

18. Sullivan Reading Read ine:;^ 

Other 

Teacher-made reading material 



Total instructional days sa^j 



PIP Speci- 
fied Core 
or Supple- 
mentary 
Material 



Core 



Supp. 
Core 
Core 
Supp. 



Supp. 



Fi equency 
of Use 



Percent 
of Days 
Sampled 



5 
5 

4 
U 

2i 
16 
12 



II 
5.7 
5,7 
4.5 
15,9 
4.5 
27.3 
18. 2 
13.6 



Brookport 



Frequency 
of Use 



23.9 
25.0 



58 
13 
18 
44 
7 

26 
2 
2 
2 
4 



i3. Grades l,and,2: .^;ith 



Percent 
of Days 
Sampled 



Primer 



OX 

0 
0 
0 
0 
0 



21.9 
4.9 
6.8 
16.6 
2.6 



0.8 
'0.8 
1.5 



46.0 



Kecugni- 
lion of 
Sounds 



Letter 



Published math material* 










1. Holt Math Test 

2. Sullivan Math 

3. Tutor Computor 

4. Singer Individualized Math 

5. Systems 80 

6. Houghton Mifflin Basic Facts & Skills 
I. Houghton Mifflin Skill Sheets 


Core 
Core 
Supp, 


2 IM 
I 1.1 
6 6.8 
1 1.1 


0 
102 
47 
23 


38.5 
17.7 
8.7 


Core 


0 0 
'} 0 
0 0 


31 
11 
24 
2 
8 


11.7 
4.2 

9,1 


8, BASE Diagnostic Test 

9. Digitor 




0 0 

0 0 


0.8 
3.0 


Teacher-ni.tde math aueriai* 




4 4.5 


6 


^.3 


Total instructional days sampled 




88 


265 



I'riraer ix 
Primary I 



Decoding 



Primary I 



Structural 
Analysis 



Primary I 
•-Advanced 



Vocabulary 



Primary Il-Advanced 



Antonyms 

and 
Synonyms 



Comprehension: 
Word, Sentence, 
and Paragraph 



Coraients 



Enjoyment, word patterns 

Enjoyment 

Enjoyment 

Enjoyment, word patterns 



Testing in all skills 



Note: Fre,.e„cy taM.Uio. »e not .de for Providence For,: «ayne City bec„e ,e W „o dat. for these grade levels; fre.ency takhtions „ere not .de for Gala, beca.e of inade.ate data, 
The skill or skills e,pfc,,si,,vJ in the »,,th .ateriah «re r..t recorded because «o were unable to find ade^Mte information on .at.rials nsed. 
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01, 



00. 



Table E-l (Continued) 



e. Grades 3 and 4; Reading 



Utch'Up Material 



Publ 1 jiied reading oaterial 



Creative realurej Structural Analysis 

Open Court Cotrchted Unguag: Arti & Reading Program 

Scholastic lndlvl('.utlUed 'te&dlng 

Sound J of Langwge Readers 

SRA Reading Program 

SRA Reading Ubot:Sj(^: 

My Own, S&ok ' 

ReaiTstory/bflok (unspecified) 

Random Houae Criterion Reading 

Syitena 80 

Oolch Vord Hat 

Hagic Seasons 

Klnlbike Fil!i;& HorkB;,«et 

?honics Use (Old i New) 

Phonics iie Use Le.'L'nin? Gameo Kit 

First Talkiog A*iph:ibet 

Specific Skills Series 

Cinn Elementary English Series 

Vords in Ho Hon 

Hiasion Read 

Core File (lessons correlated with Raadora House 

Criterion Reading) 

Cyclo Teacher 

Build A Sentence G<«ie 

Fountain Valley Reading Program I 
Singer Visual Education 
Troll Cassettes f, Fllmstrips 



Other 

Teacht-r-nude reading naterial 



Total instructional days sanpieJ 



PIP Speci- 
fied Co; 
or Supph* 
nientary 
HaterUl 



Supp. 
Core 

Supp. 



Core 
Core 



Supp. 
Supp. 
Supp. 



Frequency 
of Use 



«rcent 
of Days 
Sainpled 



5.7 
•J 
5.7 
U 
U 
2.3 
4.0 
17.1 
5.1 



17.7 
17.1 



3rookport 



Frequency 
of Use 



Percent 
of Days 
Sampled 



0.4 
38.0 
3.2 
0.7 
3»5 
0.7 
\U 
2.1 
0.4 
3.2 



0 
0 
0 
0 

3 

122 



1.1 
43.0 



Providence Forge 



Frequency 
of Use 



Percent 
of Days 
Sanpled 



11.1?. 
3.2 
2.3 

0 

3.5 



5.6 

0 



10.5 
1.5 
4.7 
B.2 



5./) 
7.6 



342 



Hayne City 



Frequency 
of Use 



d. Grades 3 and k\ Kath 



Published math naterial 

1. DrilU Facts 

2. Holt Hath Tape i Cassette 

3. Holt Hath Test 

4. SRA fUth Learning SysCem' 

5. Singer Individualized Math 

6. Sullivan Hath i Wrtkbook 

7. Systecis 80 

8. Tutor Contputor 

9. Dlgitor 

10. (lath Drawer Uorksheets 

lU Houghton Mifflin Basic Facts i Skills 

12. Nghton Mifflin Skill Sheets 

13. Milton Sradley Fractions i CassettHS 

14. Multiplication Records 

15. Creative Filmstrlps i Cassettes 

16. Drill Pages 

17. Drill Tapes 
IB. Hathlputer. 

19. Cytio Teacher 

20. SUA f.rithniedc Fact Kit 

21. DoBiinos (gane) 

22. Orbiting the Earth Came 

Other 

Teacher 'jiaile jiath material 



Supp. 
Supp. 
Core 
Core 
Core 



Total Instnictlonal days sampled 



i.n. 

5.1 
1.7 
3.4 
8.0 
6.3 
1.7 
2.9 
0 



0 0 

0 0 

0 0 

0 0 

0 0 



0 

23,4 



e.3 

26.1 
6,3 
14.8 
4.2 

I. 1 
,12.0 

II. 6 
3.5 
2.5 



0.7 

5.2 



5.0 
11.7 
7.0 



0 
0 
0 

1.5 
0,3 
1.5 



3.8 



1 

146 
4 



301 



Percent 
of Days 
Sampled 



0 
0 

6.3 

0 
0 

1.0 



3.7 
1.0 
3.7 
2.7 
2.3 
2.3 
4.7 



Primer 



Recognl* Leaer 
tlon of Recog- 
Sounds nit Ion 



15.0 



21.6 
12.6 



Mote: Frequency tabulations were m uii tor CaUx (for either reading or msth) becnuse of inadequate data. 
Sentence comprehension only. 

It was not clear in the hardvare/softvare pad'age whether the specified naterial referred to m the SRA Matn Learning Syste. or the SRA Mathematics Diagnosis. 
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Primer 
and 
Prlisrv I 



Decoding 



Primary I 



Structural 
Analysi; 



Primary l 
•■Advanced 



Vocabulary 



Primary II-Advanced 



Antonyms 

and 
Synonyms 



Comprehension: 
Word, Sentence, 
and Paragraph 



Comments 



Enjoyment 



Enjoyment^ 

Testing in all skills 



Enjoyment 
Enjoyment 



Engliah skills not listed (I.e., verbs b mm) 
Enjoyment 

Haterials covering all skills 

Testing in all skills except comprehension 
Enjoyient 



Table E-1 (Continued) 



Catch-up Material 



Published reading material 



4. 

5. 

6. 

7. 

8. 

9. 
10. 
11. 
12. 
U. 
14. 
\l 
16. 
17. 
16. 
1?. 
20. 

21. 
22. 
23. 
Z4« 
25. 

Other 



Controlled reader 

Creative Feat ires Structural Analysis 
Open Court Correlated Language Arti 
it Reading Program 
Sounds of Language Readers 
SRA Reading Laboratory: 
M}- Own Book 



Vccabulary Development 
Syatcns SO 

Random Houss Criterion Reading 
Adventure Trail 
Hagic Seasons 
Hlnlblke.Fllni Worksheet 
I>honics He Use <Old & New] 
Merrill Linguistic Readers 
Specific Skills Series 
Hlssion Read 

Clnn Elementary English Si^rles 
Scholastic Individualized Reading 
Nbrds in Hotloa 

Core File ('lessons correlated with Ran- 
dom House Criterion Reading) 
ESP Cassette Program 
Filnstrlpa-Jlm Handy 
individual Cassette Learning Package 
Language .faster Cards 
Troll Cassettes & Filmstrlps 



PIP Speci- 
fied Core 
or Sjpple* 
iiientary 
Haterial 



Supp. 

Core 

I'll r I' 



«. Grades 5 and 6: Reading 



Frequency 
of Use 



Percent 
of Days 
Sainpled 



Core 



Teacher-made reading ciaterlal 



Total Instructional days sampled 



Core 



Published math raster ial 

1. Gaises 

I Holt Hath Tapes and Cassettes 

}. SRA »«th Learning System* 

h, Singer IndlvldualUeJ |{,uh 

5. Sullivan Hath h Workbook 

6. Systems 30 

7 1 Tutor Computor 

8. BASE System 

^, HlUon Bradley Friictlpns- 6 Cassettes 

10. less Fracrlon Kit 

III Mh Diaver Worksheets 

12. Houghton Hlfflln Basic Facts i Skills 

U, Houghton Hlfflin Skill Sheets 

U. Creative Fllmtrlpa & Cassettes 

15. Hath Drills 

16. Kath Uorksheets-HiUiken 

17. Sullivan Placenent Test 

18. E<lucatlonal Activities Sew .Hath Cassettes 

19. Singer Visual Education 

20. SRA Conpu tapes 

21. Trlicore (game) 



Other 

Teacher-niade aath material 



Total instructional days sampled 



Supp, 
Supp. 
Core 
Core 
Core 



4.2?. 
2.4 

6.5 
4.2 

5.4 
3.6 
7.3 
3.6 

2.4 
10.7 
0 
0 
0 



0 

1.2 



0 

6.0 
21.4 



3.0Z 
11.3 
1.8 
7,7 
1.8 
1.8 
2.4 



33.3 



Brookport 



Frequency 
of Uee 



Percent 
of Days 
Sampled 



1.? 
29.3 
1.5 
2.2 
3.7 
12.6 
0.4 
6.3 



0.4 
60.0 



270 



Providence Forge 



Frequency 
of Use 



f. Grades 5 and 6; Hath 



5,9 
7.4 
0.7 
6.7 

4.B 
11.9 
5.9 
2.6 
19.6 
3.0 
0 



13.0 



270 



Percent 
of Days 
Sampled 



07. 
8.2 



5.3 

0 

3.5 



4.4 
3.5 



20.2 
10.0 
3.2 
3.2 
10.9 



4,7 
9.1 



Wayne City 



Frequency 
of Use 



12.9 
5.0 
1.5 



0 

0 
0 

2.3 
0.3 
0.3 
0.3 



2.6 



297 



Percent 
cf Days 



8 2.7 
97 32.7 



2.7 
1.0 



5.4 
1.7 
1,3 
1.3 
3,4 
2.0 

4.0 



01 
0 

0 

2.7 
10.4 
1.3 

10.1 



20.9 
2.0 



1.3' 
3.4 
2.0 
1,3 

10.1 

6.1 



Primer 



Recogni- 
tion of 
Sounds 



Letter 
Recog- 
nition 



Prlraer and 
Prlmay 1 



Decoding 



Primary I 



Structural 
Analysi 



Primary I 
•'Advanced 



Vocabulary 



Primary II-Advanced 



Antonyms 

and 
Synonyms 



Coraprebenelon; 
Word, Sentence, 
and Paragraph 



Comments 



Also Includes fluency 



Enjoyment 



Testing in all skills 
Enjoyment 
Enjoyment 
Enjoyment 



Enjoymcp'. 

English skills not llsii.d (I.e., nouns i verbs) 



Haterials covering all skills 
Varied jkills, usually decoding 
Enjoyment 



Enjoyment 



Note! Frequency tabulations weie not made for Calax because ol [nade(|iJato data. 

*Tli(! hardware/software package did not clearly indicate whether the mtorial referred to vas th. SRA Hath Learning System or the SRA Hathemfltics Diagnosis, 
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Table E-1 (Concluded) 



g. Grades 7 and I 



Catch-Op Material 



PIP Speci- 
fied Core 
or Supple- 
mentary 
Material 



Wayne City 



Frequency 
of Use 



Percent 
of Days 



Primer 



Recognition 
of Sounds 



Letter 
Recognition 



Pricier and 
Primary I 



DecodiriK 



Primary I 



Structural 

.s 



Primary I 
"Advanced 



Vocabulary 



Primary II-Advanced 



Synonyms 



Comprehension: 
Words, Sentence, 
and Paragraph 



Coments 



Published reading material 

1. Cycle Teacher 

2. Frob<i (game) 

3. ESP Cassette Program 

4. Fountain Valley Reading Program 

5. Media Cassette 

6. Random House Criterion Reading 

h Core File (lessons correlated with Random House 

Criterion Readinj?) 
a. Sc'jlastic Skill. Books 

9. Troll Cassettes 6 Filinstrips 

10. Tufabet Vocabulary Building Game 

Other 

Teacher-made reading material 



Core 



4 
3 

1 
k 

% 

32 

3' 
5 
2 

3 

2& 



1.4'/. 
1.0 
2.1 
2.4 

I. 4 
33.3 

II. l 
1.0 
1.7 
0.7 

1.0 

9.0 



Varied skills, usually decoding 
Testing in all skills 

Testing in all skills 

Materials covering all skills 
Enjoyment 



Total instructional days sampled 



h. Grades 7 .indS: Math 



Published m:' Mterial 






1, Hought'jii Mifflin Ba^ic Facts i Skills 




52 18,17. 


2, Hath Facts Division Game 




2 0.7 


3. Domlnos' tganic) 




4 1,4 


4. Singer Individualized Hath 


Supp. 


5 1.7 


5. Singer Visual Education 




6 2.1 


6, Spinner Number Games 




4 1.4 


7, SRA Conip-j tapes 




6 2.1 


8. Sullivan Hath 


Core 


28 9.7 


9. Triicore (game) 




5 1.7 


10. Tutor Computer 


Core 


13 4.5 


Other 




15 5.2 


Teacher-mde math mtmA 




22 7.6 


Total Instructional lays sampled 




2bJ 



Note! Frequency tabulations were not made for Bloonilng^on, Bruokpor' , or Providence Forge because these sites did not have students participating in the program at these grade level: 
frequency tabulations were not Tiade for Galax because of inadequate data. 
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Table E-2 

MOST FREQUENTLy USED MATERIALS AND SKILLS EMPK^IZED IN CONQUEST PROJECTS, BV GRADE LEVEL 
a. Grades 1 and 2; Reading 



UtKjuesC Material 


riP opeci' 

fied Core 
or Supple* 
mentary 
Material 


Benton Harbor 


Cleveland 


Primer 


Primer and 
Primary I 


Primary I 


Primary I 
"Advanced 


Primary II~Advanced 


Comments 


Frequency 
of ^se 


Percent 
of Days 
Sampled 


Frequency 
of Use 


Percent 
of Days 
Sampled 


Recogni- 
tion of 
Sounds 


Letter 
Recog- 
nition 


Decoding 


Structural 
Analysis 


Vocabulary 


Antonyms 

and 
Synonyms 


Comprehension; 
Word, Sentence, 
and Paragrapii 






























1. Gaioes 




?fi 

^0 




U 


















Varied skills 


1- D.C. Healh (lork shops /Dookshops 




54 


19.3 


0 


0 




X 


X 




X 




X 


3. Merrill Reading Skill Text 


Core 


100 


35.7 


0 


0 


X 


X 


X 


X 


X 








4, Phonovisual Phonics 




79 


28.2 


0 


0 


X 


X 


X 


X 










3. Patterns, Sounds & Meaning 




48 


17.1 


0 


0 


X 


X 


X 












6. r.*ad story /book 

1, Phonovisual Consonant Workbook 




52 
32 


18.6 
11.4 


0 
0 


0 
0 


X 


X 


X 










Enjoyment 


■8. McGraw-Hill Progra-raed Reading 


Core 


201 


71.8 


36 


15.5 


X 


X 


X 


X 


X 








9. Systems 60 


Core 


58 


20.7 ■ 


64 
82 


27.5 


X 


X 


X 


X 


X 








10. Dolch Vocabulary Words (WCRC)^ 


Core 


b9 


24.6 


35.2 










X 








11. Bowiar Priniary Reading Series 


Core 


0 


0 


70 


30.0 
















Enjoyment 


12. Reader's Digest Individual Skill Builders 


Supp. 


0 


0 


54 


23.2 














X 




13. Specific Skills Series 


Core 


0 


0 


44 


18.9 


X 








X 




X 




14. Ideal Tapes i Worksheets 
13. Phonics We I'se 




0 
0 


0 
0 


31 
39 


13.3 
16.7 


X 


X 


X 

X 


X 








Varied skills, usually decoding 


16. Reati, Study, Think 


Core 


0 


0 


19 


8.2 


X 












X 




17. Steck-Vaugho Individualized Directions in Reading 




0 


0 


16 


6.9 






X 


X 






X 




Other 




138 


49,3 


95 


40.8 


















Teacher-made reading material 




16 


5.7 


155 


66.5 


















Total instructional days sampled 




280 


23) 


















b. Grades 3 and 4: Reading 


Published reading material 




























I. D,C. Heath Workshops /Bookshops 




50 


20.87. 


0 


0 I 




X 


X 




X 




X 




2. Merrill Reading Skill Text 


Core 


61 


25,4 


0 


0 


X 


X 


X 


X 


X 








. 3. ht terns, Sounds & Meani^ 




44 


18.3 


0 


0 


X 


X 


X 


' X 


X 








4. Phonovisual Phonics 




49 


20.4 


0 


., 0 


X 


X 


X 


X 


X 






Encoding as part of decoding skills 


3. Language Master Cards 
6, SRA Reading Laboratory 


Core 


25 
30 


10.4 
12.5 


0 
0 


0 
0 






X 


X 


X 

X 


X 


X 


Varied skills, usually decoding ^ vocabulary 


7. Specific Skills Series 


Core 


% 


23.3 


89 


33, J 


X 








X 


X 


X 




8, McGraw-Hill Progranmed Reading 


Core 


181 


75.4 


48 


18,2 


X 


X 


X 




X 








9. Systems 80 


Core 


45 


18. 8 


22 


8,3 


X 


X 


X 


X 


X 








10. Dt. Spello (WCRC)^ 


Core 


34 


M 


4 


1.5 


X 


X 


X 


X 


X 








11. Dolch Vocabulary Words (WCRC)^ 


Core 


0 


0 


24 


9.1 










X 








12. Read, Study, Think 


Core 


8 


3.3 


16 


6.1 


X 












X 




13. Scholastic Skills Books 




0 


0 


43 


16.3 


y 


X 


X 


X 










14. Phonics k'e L'sr 




0 


0 


54 


20 5 


X 


X 


X 


X 










13. Controlled Reader 


Core 


1) 


0 


21 


/.8 










X 


\ 


:< 


Also ircluJes fluency i speed 


16. Merrill Phonics Skill Text 


Core 


0 


0 


16 


6.8 


X 


X' 


X 


X 


X 






17. Reader's Digest Ldividual Skill Builders 


Supp. 


0 


0 


83 


31.4 












X 




Other 




155 


64,6 


10^'. 


39,4 














J " 




Teacher-made reading material , 




48 


2C,0 


206 


78.0 


















Total instruct ionril days sampled 




2^0 


264 



















Note; Frequency tabulations could not be mv: for Glovfrsvi.lc because of inadequate data. 
It '*vis uncU'.u t.'hfit likills vcre covtri^d duv to lack of knowledge about ganos used. 

WCRU = Webster Classroom Reitdin^ Clinic, vhich includes; Conquests In Reading, Dr. Spello, Dolch Basic Sight Vocabulary, and others. TTie components yere de.ilt with Individually beciusc of the different skill'] thnt each covered. 



Table B«2 (Concluded) 
c. Grades 3 and 6; Reading 



Conquest Material 



Pubhshed reading material 

1. Games 

2. D.C. Htsth Workshops/Bookshops 

3. Merrill Reading Skill Text 

4. Patterns, Sounds i< Meaning 

5. New Spelling Goals 

6. SRA Reading Uboratory 
/. Specific Skills Series 

8. McGraw-Hill Programed Reading 

9. Systems 60 

10. Merrill Phonics ikill Text 

11. Reacltii's Digest "nriividuai Skill MU 

12. br. Spello (VCRC'^ 

13. Conquests in Rwding {WCRC)t ' 

U, Styck4aughn Indvidualized Directimf i'. He.j'i ■ 

15. Pho'ilcs We Use 

16. Ta^'.h-X 

OthKr 

Teacher-made readinj; miiterial 



PIP Speci- 
fied Core 
or Supple- 
mentary 
Hate -la 1 



Beiiton Harbor 



Frequency 
:f Use 



Percent 
of Days 
Sampled 



ire 

■ore 
;ore 



;pp. 
jre 
Core 



To;al instructional days sampled 



20 
57 
30 

21 
39 
35 
164 
^1 
k 
k 

f)4 
19 



1^2 



25.0 
13.2 
43,4 
9.2 
17,1 
37.3 
71.5 
13.6 
l.B 
1.3 

:j.i 

3.3 

0 
0 



Cleveland 



Frequency 
of Use 



0 
0 
0 
0 

93 
?0 
73 
90 
19 
28 
54 
47 
35 
24. 

150' 

21 



m 



Percent 
of Days 
Sampled 



0 I 



0 

3j.8 
3:-. 7 
]6.5 
32. 7 
14.2 
10.2 
19.6 
17.1 
\ll 
3.7 

54.5 



FrlQer 



I Primer and 
I Primary I 



I 275 



tion of 
Sounds 



Letter I 



Decoding 



nit ion I 



Frlmery I 



Structural 
Analysis 



Primary I | p^jj^^^y ;i-,Advaiiccd 
"Advanced i 



Voca'uulary 



I Antonyms 



Synonyms 



Comprehension' 
Moid, Sentence, 
and Paragraph 



i/arled ikula 



Encoding as part oi decoding skills 



Also includes fluency & speed 



M( riL^^uency tahulations could nut bo made for Glovursville because of inadequate data, 

i< 

It wjs unclear what skills wiire ■ wered due lack of knowledge a.oi't ^^anes used. 

\'aC iJebster Classroom Rt^ading Clinic, which includes: Conquests Rscding, Dr. Spello, Dolch Dasic Sight Vocabulary, and othtis. The co'.i,ponent.'' weru dealt with individually K.causc of the diffcr.^n: skills that aach covered. 
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Table E-3 

MOST FREQUENTLY USED MATERIALS AND SKILLS EMPHASIZED IN HIT PROJECTS, BY GRADE LEVEL 
a, Grade 6: Reading 



HIT Material 


PIP Speci- 
fied Cote 
or Supple- 
mentary 
Material 


Lexington 


Primer 


Primer and 
Primary I 


Primary I 


Primary I 
--Advanced 


Primary II-Advanced 


Coiranents 


Frequency 
of Use 


Percent 
of Days 
Sampled 


Recognition 
of Sounds 


Letter 
Recognition 


Decoding 


Structural 
Analysis 


Vocabulary 


Antonyms 

and 
Synonyms 


Comprehension: 
Word, Sentence, 
and Paragraph 


Published reading material 

L Adventuring in the City 
■ 2. Conqu-^ts in Reading* 

3, Games 

4, Phonics We Use 

5, New Phonics We Use 

6, Read story/book (unspecified) 
]. Remedial Reading Drills 

8. Stories of the Inner City 

Other* 

Teacher-made reading material 


Core 
Core 


2 m 

34 18.8 

10 5.5 

11 6.1 

3 

2 hi 
62 34.3 
20 11.0 

4 2.2 
16 8.8 


X X 

X X 
X X 


X 

X 
X 

X 


X 

X 
X 


X 


X 

X 
X 


Enjovment 
Varied skills^ 

Enjoyment 

EnjofiK'nt 


Total instructional days sampled 




181 



Note: Frequency tabulations were not made for Clean because it did not have students participating in the program at this grade level. 



This material was included in the hardware/ software packet by mistake, but Lexington continued to use the material during the second year of the program, because they had 
problems obtaining the Sullivan Programed Reading materials. 

^ 

'It was unclear what skills were covered due to lack of knowledge about gamea used, 

Includes Pay Day activitits; Pay Day involves use of a bank book, which is not a published material but is specified in the PIP. 



Table E-3 (Concluded) 

b, Grades J-9: Reading 



HIT Material 


PIP Speci- 
fied Core 
or Supple- 
ments rv 
Material 


Die 


3n 


Lexington 


Primer 


Primer and 
Primary I 


Primary I 


Primary I 
"Advanced 


Primary II-Advanced 




Frequency 
of Use 


of Days 
Sampled 


Frequency 
of Use 


Percent 
of Days 
Sampled 


Recognition 
of Sounds 


Letter 
Recognition 


Decoding 


Structural 
Analysis 


Vocabulary 


Antonyms 

and 
Synomyms 


Comprehensicn: 
Word, Sentence, 
and Paragraph 




Published reading material 




























1, Adventuring in the City 

2, Conquests in Reading 

3, Phonics We Use 




0 
0 
0 


0 I 

0 

0 


9 

36 
19 


4.37o 

IJ.I 

9.0 


X 


X 
X 
X 


X 


X 




X 




Enjo}'ment 


4. New Phonics We Use 




0 


0 


5 


2.4 


] 


X 


X 




X 






5.. Sr.ories of the Inner City 

6» Gaaies ,* 

h Remedial Reading Drills 

8. SRA Reading Laboratory 

9. Sullivan Comprehension Readers 

10. Sullivan Programmed Reading 


Core 
Core 

Co'ce 


0 
4 
38 
5 

64 
9J 


0 

2.6 

24.7 
3.2 
41,6 
63.0 


21 
3 

78 

0 
0 
0 


10.0 
1.4 
3J.1 

0 
0 

0 


X 


X 


X 
X 
X 


X 

X 
X 




X 

X 


X 

X 


Enjoyment 
Varied skills* 


Other 




8 


5.2 


at 


3.8 


















[eacher-made reading material 




0 


0 


8 


3.8 


















[otal instructional days sampled 




154 


210 



















c. Grades Hath 



Published math material 




Olean^ 


1. Self-Teaching Flashcards in 
Addition and Subtraction 

2. Self-Teaching Fli^hcards in 
Division 

3. Self-Teaching Flashcards in 
Multiplication 

4. SRA Arithmetic Fact Kit 

5. Sullivan Math5 

6. Pay Day 






Core 
Core 
tore 
Core 


20 Ml 
46 24.0 

56 IVi 
1 1.0 

160 63.3 
3 1.6 


fotal instructional days sampled 




m 



It was unclear what skills were covered due to lack of knowledge about games uscl, 

Includes five Pay Day activities; Pay Day involves use of a bank book, which is not a published material but is specified in the PIP, 
Frequency tabulaMons were not made for Lexington because of inadequate data relative to math instruction. 

Although the Sullivan Math Program is not specified in the hardware /software packet, the Sullivan Placement Test and Test Booklet are specified and are part of the Sullivan Hath Program. 



Table l-k 

HOST FREQl'miV USED MATERIALS AND SKILLS EMPHASIZED IN IRIT PROJECTS, BY GRADE LEVEL 
a. Grades 3 and 4: Deciding Instruction 



PIP- 

Specifled 
Core or 
Supple- 
mentary 
Material 



Sup?. 



Core 



Bloonlngtoii 



Frequency 
of Use 



Percent 
of Days 



m 



6.9?. 
15.6 
22.0 
55.5 

4.0 
16.8 

6.9 

fl.7 

32,9 
19,/ 



Oklahoma City 



Fiequency 
of Use 



Percent 
of Days 
Saispled 



0 
0 

14 

63 
1/ 
19 
43 
161 
50 

3:- 

58 

0 



11'} 



7.6 
8.4 
19.1 

n.6 

22.2 
14.2 
25.8 

0 

0 

0 



8.9 
3,1 



Schenectady 



Frequency 
of Use 



Percent 
of Days 
Sampled 



0 0 



L.2 



9.4 



0 
0 

0 

5,9 
5.9 
5,9 
5.9 
12.9 
17.6 
11.8 
16,5 
9,4 

23,5 

0 



Primer 



I Primer and 
I Primary I 



tion of 
Sounds 



Letter 



Decoding 



nilion 



Primary I 



Structural 
Analysis 



PriiDary I 
"Advanced 



Vocabulary 



Primary II-Advsnced 



Antonyms Comprehension: 
and Word, Sentence, 
Synonyms and Paragraph 



Comments 



E.ijoyraent, listening skill; 



Enjoyment L fluency 

Viiiit'd skills, usually 
vuciibulary t» decoding 



Testing in all skills 
Testing in all skills 

Enjoyment 



ERIC 



Table E-4 (Continued) 
b. G. ' nd4: Individualized Reading Instruction 



IRIT Reading Material 
(Individualized reading instruction] 



fubl 



.shed fltdierl?! 

1. ilesr s Picnic 

2. Frances Scries 

3. Walt Disney Story Records 

k, Individualized Cassette Learning Package 
5. 
6. 



Scholastic Individualized Reading 
Specific Skills Series 
Tape Recorder uith Books 
Trade Books 

Work gfjding 'Jith Specific SI;iIU Series 
ii Random Hruse Reading Materials^' 
Auto Vance Films 

"ontinuous Progress Laboratory for 
Lan-jage Arts 

Hoi.man Language Arts Reading Program 
Language Master Card Program 

Newspaper Stories 
Adventures in Glen Scries 



Dr. Scuss Series 

Mr. 5t Hrs. Bumbo Series 

Recreational Reading 

Taylor fibstrips; Tell-He-A Story Library 



Other 

Teachernnade reading material 



Total instructional days sampled 



PIP- 

Speclfied 
Core or 
Supple- 
mentary 
Material 





11 

7 


4.4Z 
2.8 


Core 


13 


5.2 




25 


lO.O, 


Core 


' 127 


50.8* 


Core 


90 


36.0 


Core 


0 


0 



Supp, 



frequency 
of Use 



Percent 
of Days 
Sanipled 



0 0 



138 
52 



55.2 
20.8 



250 



Oklahoma City 



Frequency 
of Use 



36 
39 

75 
156 
13 



Percent 
of Days 
Sampled 



2,2 
13,3 
34.2 
52,4 

16.0 
17,3 

33.3 
69.3 
5.8 



24 10.7 



3.6 



225 



Schenectady 



Frequency 
of Use 



Percent 
of Days 
Sanipled 



23.2 
ILO 
14,6. 
11.0 
7.3 

32.9 
1.2 



Priroer 



tion of 
Sounds 



Letter 



nit ion 



Work grading is^:pot a published material buc is a specific and separate use of the published i!iateri?ls listed, 
Word and sentence c(taprehension only. 



Primer and 
Primary 1 



Decoding 



Primary 1 



Structural 
Analysis 



3vi i ^^'^'^y ^^-^M 



I Antonyms 
Vocabulary | and 
I Synonyms 



Comprehension: 
Word, Sentence, 
and Paragraph 



Connts 



Enjoyment 
Enjopent 

Enjoyment, listeiiing skills 
Enjoyment, listening skills 



Enjoyment, listening skills 
Enjoyment 



Enjoyment, [isuo.rj, iiUs j 



Varied skills, usually de- ! 
coding ^ vocabulary j 
Writing skills j 
Enjoyment ; 
Enjoyment, listening skills | 
Enjoyment I 
Enjoyment 
Enjopent 

Sequencing, listeninf.-i & 
speaking skills 



ERIC 



0' 7 
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Table S-^ (Concluded) 
c. Grades 3 and 4; Vocabulary and Comprehension 



IHIT Reading Material 
(vocabulary anti coraprehensivn) 



Published material 

1. Bowniar Highway Holiday Se-ies 

2. Continuous Progress in Spelling 

3. Random House Criterion Reading 
PhooniK Reading Series 

5. P..,is 

6. Story Picture 

/, Controlled Reader 

fi. Reader's Digest Individual Skill Builders 

9. Specific Skills Series 

10. SRA Reading Laboratory 

11. Houghton Hifflin Readers 'i Vorkbook 
il, Language Center 2 

n. Jlacniillan Sasals *i Workbook 

li. }1erriU Linguistic Readers 

15. Taylor Filrastrips; Tell-Me-A Story Library 

16. Typing-HcCrdw-Hill 
Other 

rescherisade reading material 



Total instructional days sainplec! 



PIP^ 
Specified 
Core or 
Supple- 
mentary 
Material 



Freijucncy 
of Use 



Supp, 
Core 
Core 
Core 



Core 



Percent 
of Bays 



Oklahoma City 



Frequency 
of Use 



26,21 



10.4 
9.3 



Percent 
of Days 
Sampled 



225 



24,9 
25,8 
4,4 
21,3 
11,6 
19,1 
36.0 
8.4 

34.2 
19.6 
15.1 



Primer 



I Primer and 
' Primary I 



Recogni- 
tion of 
Sounds 



Note; Fretjuenty tabulations could not be made for Schenectady because no data were available. 
Hcrci and sentence comprehension only, 



Letter i 



nitioni 



Decoding 



Primary 1 



Structural 
Analysis 



Primary I ' 
-"Advanced I 



Primary II-Advanced 



Vocabulary 



Antonyms 

and 
Synonyms 



Comprehension: 
Word, Sentence, 
and Paragraph 



Cciinents 



Enjoyment 

Testings in all skills 

Enjoymtnt fi word 
Enjoyment 



Sequencing, listening skill.^ 
h speaking skills 
LaiiguBge arts skills 




C I 



Appendix F 
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Appendix F 



MAT SUBTEST ITEMS USED IN REGRESSIf ^^ ANALYSES 

Mat su'btest items displayed in Table E-1 are lghvv . .^ble between MAT 
batteries shown. At the t'. v and eighth grades, ' ii 'ems compared do 
not represent the total pos.*.. "e comparable items, hr the compar- 

able items among those selected relevant to the /iT -^ur :: : cui im only. 



Table ^- L 

PARALLEL MAT SUBTEST LTHMS RELEVANT TO PIP CURRICULUM 













Word Knowledjje 


Subtest : 




All Subrosts: Graco 1 Only 


All Grades ExceoC Grad-* I 


(a) 


■ 

Primer: L-istening 


Prima 


ry I: 


(a) 


Primary I 


Primary il 




f'- r Sounds 


Wrrd Analysis 










34 


4 








14 




36 1 


2 






32 


11 




37 


21 












39 


5 




(b; 


Primsi'y II 


Elementary 


(o) 


Road ing 


Word 


tCnowled",e 




None 


None 




16 


3 




(c) 


Elementary 


Elen.entary 




17 


5 






1-5U 


1-5C 


(c) 


Rend ing 
None 


Re ad 
I . J 




(d) 


Elemenca--y 
None 


i ntermedialG 
Nc'ie 


(ci) 


Numbers 


Mache 


: aCics 


(e) 


^iiCerrr^ediate 


Advanc';d 




I 


•> 
J 






None 


!-'')ne 




3 


2 












4 


7 






HIT Advanced 


HIT Advanc-d 




5 
6 


.0 
12 






1.-50 


L - iU 




8 

10 


4 
32 






R-3 Advanced 


R-'^ Advan.;cd 




12 


33 








1-50 ■ 




13 
14 


9 




t 
i 








17 


17 












19 


16 












21 


39 












->■> 


38 












2.3 


36 














40 




l» 








25 


41 












26 


48 












27 


49 












28 


50 












30 


37 












31 


43 












32 


44 












34 


47 











3V3 



Table F-L (Continued) 



Re^dinR Subtest: ALL Grades Except Grade I 



(a) 


Primary I 


Primary II 




Elementary 


i 

Intermediate 




29 


15 




29 


5 




30 


LA 




30 


10 




31 


16 




31 


8 




32 


18 




32 


7 




38 


25 




34 


9 




39 


30 




36 


3 




40 


26 




38 


I 




42 


29 




39 


4 










40 


24 


(b) 


Primary II 


Elementary 




42 


22 




22 


14 




43 


28 




23 


11 




45 


26 








(e) 








25 


i 


Intermediate 


Advanced 




26 


3 




16 


20 




29 


4 




18 


22 




36 


5 




19 


21 




37 


6 




36 


13 




38 


7 




39 


14 




39 


8 




40 


11 




40 


9 




41 


5 




42 


18 




42 


7 




44 


17 




43 


6 










44 


8 




Elementary 


E lementary 




45 


9 




1-2B 


L-28 


(f) 












HIT Advanceti 


HIT Advanced 










1-15 


1-15 










30-37 


30-37 








(g) 


R-3 Advanced 


R-3 Advanced 










"l-l5 


1-15 










30-37 


30-37 



F-5 





j,. 


K- 1 


s I 1 


t i nu 








C'-t:idu I •. .. I on ^ 




ALL 


GiMtk-.-: Kxcept Grade 1 


( n") Prirnary 


1 ; ? r ir.Mry 


1 


u!) 


K Lemc-iuar y 


In rciTinod ia to i 


>Li Lho'.na Lies, 










1 

•5 i 


Part B 




i 




LI 




9 ' 

i 




L'J 






L2 




6 i 


•J8 


L 






L4 




22 j 


•J9 


■1 






17 




4 i 


41 








9 9 




5 


4L! 


J 


i 




24 




3 i 


^6 


['2 


i 




27 




19 


48 




1 




23 




7 


50 


8 


1 




29 




17 


53 


■ 5 


i 




30 




L'3 


55 


in 


1 
1 




3L 




I 


56 


29 






34 




34 


57 


6 


1 




36 




L8 ' 


58 


14 , 


1 

1 




40 




11 


.u 


I i") 

I 5 






Ln lurmcd late 


* • vj w n 1 1 \. v.. vj 


1 
1 








5 




1 


1 (b J PriTt.'irv 


a ry 




LO 




5 


1 , 


o 






IL 




6 


i CS 


10 






12 




L4 


i ' 


A 






13 




4 


i LO 








15 




2 


1 a 








L^i 




9 


I n 








20 




7 




1 L 






23 




12 


\ ^ 


3 






?. » ) 
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Table F-l (Continued) 
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ANALYSIS OF CORRESPONDENCE BETWEEN THE MAT 
AND rOURTli AND EIGHTH GRADE CURRICULA 



Appendix G 



ANALYSIS OF CORRESPONDENCE BETWEEN THE MAT 
AND FOURTH AND EIGHTH GRADE CURRICULA 



Our anal],ses showed that at a gross level the PIP curricula covered 
skills tested by the MAT. However^ these analyses are not entirely sat- 
isfactory because they were based on global judgments^ without regard to 
the placement of students relative to the curriculum materials. In this 
section we report analyses that are more conservative than those re- 
ported above^ in the sense that we are much more careful about speci- 
fying what we mean by a skill being ^'tested by the MAT." 

To determine if the MAT is truly testing the curriculum, we must be 
sure that the skills needed to answer an item correctly are "taught" in 
the curriculum materials. We must also be sure that we have criteria 
for deciding that students covering those curriculum materials would 
have learned those skills. 

Information was available in the SOIs at a fairly fine level of de- 
tail on what lessons students had covered. We therefore had a reason- 
ably detailed picture of what materials were used in teaching^ but could 
not say exactly what was taught because materials could be used to teach 
several points. We hoped that one of the points taught was the one that 
the author of the iroterials intended; if so^ we could much, more effec- 
tively assess the relevance of the MAT to PIP outcomes^ and could re- 
strict our statistical analyses to just those items that were relevant 
to the curriculum. 

Our objective was to establish a correspondence between the PIP- 
specif ied-and-used curriculum and the MAT^ based on our idea of what 
skills were necessary to learn an item (i.e.^ to answer it correctly). 
Mapping this correspondence entailed two tasks: (1) analysis of the 
skills required for a correct response to each test item^ and develop- 
ment of the rules or criteria for deciding what in the curriculum would 
be an exemplar of that set of skills and (2) development: of procedures 
for searching the curriculum materials to find units that satisfied the 
rules. In this section^ ther.^ the methods for determining congruence be 
tween what was tested and what was taught are presented. 



Our use of tiiese methods was limited to the Elementary and the Ad- 
vanced levels of the MAT and to those PIP projects with students in the 
fourth and the eighth gr.^des (except for R-3). Only at these grades 
were the children tested in both the fall and spring on the same form 
an..' level of the MM. We did not distinguish between the speci f ied-and- 
us-.:a curriculum of fourth grade Catch-Up, Conquest, and IRIT because they 
were about the same* Thtr oighuh grade curriculum is the same as that for 
HIT, 

Skills Analysis of the Elementary and Advanced MAT 

Analysis of the MAT. began vith an examination of each item. Our 
strategy for determining what skills would be necessary for answering 
that item correctly v/as to ask ourselves v/hat would ''teach" that partic- 
ular item as presented in that particular format. Generally we found 
that each test iteii required a combination of skills including re- 
sponding appropriately to the item format. We tried to imagine the kind 
of curriculum unit that would give a student experience with the set of 
skills and knowledge he would need to ensure a correct response. 

In formulating the rules for determining whether the test item had 
been "taught/^ we v;ere extremely literal about the features of the item. 
We included in our rules, or criteria for declaring a match between cur- 
riculum and test item, all the features we felt were essential for the 
student to answer the item correctly. We wished to take this conserva- 
tive approach so that^ when eventually we included a test item in our 
analysis of project effectiveness^ we could assert that students covered 
the materials appropriate to passing that item. 

Word Knowledge Subtest 

The rules for the Word Knowledge subtest were quite unambigu- 
ous. Gonerally^ for each test item^ the two words in tlie stem of the 
item and che correct answer word had to be foi^nd in tlie ci^rriculum for a 
match to be declared. (For example. Item 3 iu Che Elementary subtest is 
"happy means glad." In the stem of the item^ "happy" is the target word^ 
and "meaii.^" is the context world. The answer word is "glad.") Modified 
versions of any of the three words could not change the meaning of the 
vord. We used the principle of "near transfer" as a guide to limit 
which modifications would be acceptable."' For example^ "dependable" and 
"depend" v/ould be considered acceptable because the idea of being reli- 
ant on another is basic to both x^ords. "Please" versus "pleasing" w^ould 
not be credited because "please" usually functions as a way to express 
politeness^ whereas "pleasing^' cornotes giving pleasure. 



This principle as.^umes hardly any generalization of the skills. 

3dJ 



We felt that all three.-.words- in an item ^re important because 
to: .get to the answer or critical word from the target word the student 
must understand the context, word... For example, to know that "night" is 
the. opposite of "day," th6 student must know the meaning of the context 
word "opposite." We therefore concluded that the stuc^^nt must have had 
experience with all thpee words.. \ 

•. . For the Word Knowledge subtest we claimed that\ a curriculum 
unit taught a word only if thf particular word and its m^^ning were 
treated in a well-marked exercise and if several practice\items were in- 
cluded in the exercise. It was not practical to set a thr^eshold,^ such 
as two or four practice it^ within an exercise,- because d£ the diver- 
sity, of presentations, ia- the curriculum materials. For word knowledge 
items, it was also necessary that the meaning of the word be\singled out 
for attention in the curriculum unit ot exercise and that theVstudent be 
required to determine the meaning of . the word from some conte:^^ual clues. 

Reading Subtfeft \ 

In the Reading subtest, a pupil is required to read a passage 
and resp.ond to several questions about the passage. Exhibit G-1 it an 
example from the E 1 erne ntary .MAT . Rules for MAT items were based ol two 
kinds of features-one describing the passage and the other specifying 
■the type of question, posed about the passage. Although we believe kViowl- 
edge of the content of a passage would sometimes permit the item to l\e 
answered correctly even when the passage was not read or comprehended^. 
(Tuinman, 1973-74), we could not conceive of any way to develop rules A 
for matching content in, the test passages : with content in the curriculum 
Because of our conservative strategy we were not particularly concerned\ 
with whether students could pass items without having covered relevant • 
curriculum material. Rather we^ wished to claim that certain curriculum- 
materials contained all the requisite skills for certain items and thus \ 
that students who could not formerly pass the item should now be able \ 
to pass. ' 

Passages used to test reading comprehension vary on several di- 
mensions besi^^o content. They are generally made more difficult to com- 
prehend by (1) containing more words that either occur infrequently in 
the students' experience or are abstract or complex in meaning, (2) em- 
ploying more phrases or clauses requiring ideas to be temporarily stored 
in memory before the message is complete, or (3) lengthening the passage 
so thatattentional skills, memory, or search skills are taxed. The MAT 
publishers stated t^a^t variation, in the reading passages occurred along 
three dimensions : vocabulary level, syntactic complexity, and length of 
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EXHIBIT. SAMPLE PASSAGE AND ITEMS FROM tHE ELEhENTARY MAT 



Each' year on November 5, people in England 
celebrate a special, holiday. The holiday, Guy 
Fawkes Day, is enjoyed by both children and 
older people. Huge bonfires are lit, and in the 
evening,^ children" set off fireworks. Fawkes 
lived' more than^^50 :>ears ago. He took part 
in the famous "Gunpowder Plot" against the 

^government 'The English still celebrate the day 
because . the plot" was discovered before anyone' 

* was hurt. • 



17 The bcsi name for this story would be 

XD Holidays 
CD An English Holiday . 
O The Life of Guy Fawkes 
® Bonhres 



. i« Children set off fireworks—. 

CD all day 
■ CD on bonfires 
CD after November 
CD inUhe evening 



1' Instead of still, in the last sentence, you could 

say— : . 

' .CD calm ■ ; 
CD quietly 

■CD. continue, to . • ' 

CD at rest ■ 



20 



.The gunpowder plot probably took places- 

(T^ - tn Rritich l»»o»»hH 



O vin British legend 
O on a holiday 
O in the spring 
CD in the early . 1600's 



Reproduced from the Metropolitan Achievement ?Jests, 
.copyright © 1970,. by Harcourt Brace "Jovanovich,. Inc. 
Reproduced by jpecial permission of the publisher. 
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• passage. We quantified these dimensibns to cre'ate rules for matching 
MAT test passages with PIP curriculuiji passages,' . . - 

The Spache and Dale-Chall/ readability formulas (Spache; Dale 
and Chall, 1948) were used to measu're the first dimension (i.e./ vocabu- 
lary level). These formulas generate a quantitative measure in the form 
of a grade level for which vocabulary is appropriate. The indices are 
primarily dependent on vocabulary/ bun include some adjustment for aver- 
age sentence length. in the pac-sa^e. The number of words in a passage 
thatare not included on a maste/: list are counted. The greater the num- 
ber of exclusions, the more difficult (higher grade level) the passage 
IS rated. The Spache index was/ used for passages' up to 4th grade level 
and the Dale-Chall index was used for 5th -tp Uth grade material. The ' 
Fles;ch formula (see Klare, 197/4-75) has been used for fourth-fifth grade 
but Its reliability has been (Questioned. Therefore, a weighted combina-' 
tionof the Spache and Dale-c/hall formulas wasused to quantify readabil- 
ity at approximately the foytth to fifth ^ra'd^ level.* 

^. • These two fopnula4, among the numeroiis readability indexing 
techniques, predict reading level most reliablyX (Klare, 1974-75)^' The 
test publishers used a simple noun count, ;l.ut this technique .underesti- - 
rnates difficulty (Klare) . A" computer program (Juad,M975) was used for ' 
this measure to reduce tedious hand calculations. .. \ ^ 

. For syntactic complexity- the second dimension of passages to 
be indexed-a sensitive measure was considered. This measure, the unit 
of which is called a T-unit (an independent' and linked dependent clause) 
has. accura.ely, discriminated, among children's writing as well as reading' 
passages in norm-referenced tests. (Hunt, 1965; Calfee, 1975). Another 
cluster of syntactic dimensions, including occurrence of certain syntac- 
tic features in words (Golub and Kidder. 1974),. was also examined. Both 
techniques were rejected because-, the amount of readirife materials in the 
PIP curricula was too extensive to analyze hf these methods within the 
time frame of tiie projec". 



Tills weighting was constructed by using the length of the passage (say 
63 words) as a percentj>> to adjust the Dale-Chair (say 9..5 grade level). 
The product (grade level) was averaged with the Spache grade level. The 
caveat appropriate foi i.ever averaging grade level equivalent -test . scores 
does not apply to these grade level scores. The grade level scores in 
readability .•:>rmulas are not projections of performance; they character- 
ize mc^.^rials. However^ like grade level equivalency scores, they are 
viewed as approximations, indices of a vocabulary level of reading com- 
prehension. . 



A ^simple measrure of sentence length (number of ^ords in . pas- 
sage divided by number of sentences in passage) was chosen to reflect 
synjiactic complexity. The longer the sentence^ it was assumed, the more 
.cognitive processing is required to understand it. This assumption, 
while. weaker than we would have liked, "holds up frequently enough to jus- 
ti fy its use« ■ . / 



^ Length . of the passages that -the student must read was the 
third dimension, or feature, measured. In most tests, length of a pas- 
sage discriminates among^students ' test scores because the test time is 
... limited and students who read faster cover more items . ' Because the stu- 
dent rarely encounters severe ttme limits in the classroom, the, test is 
measuring performance for which he' has not been trained. In addition, 
his skills cannot be tapped if he is unable to finish the long passages 
due to time constraints. Consequently, the length of each passage mea- 
sured,-. in number of words was viewed ks a critical feature. 

"^^^^ passages had different combinations of the three measures. 
Sometimes, a long passage ha' a lower vocabulary level than a shorter 
passage, or a short passage contained long sentences. Our purpose in 
analyzing the test required that the three quantitative scores for each ' 

■ MAT test. passage be compared with the corresponding scores for each sam- 
ple of the PIP curri.culuik. I Table G-1 shows the ranges on the three mea- . 

• sures foi:-^- matching PIP curriculum passages with the Elementary and Ad- 
vanced -levels of the MAI. /The passages were matched when a PIP 
curriculum unit had scores that fell within the same ranges as the MAT 
testVpassage on all. three Measures (or within the, same range, on two mea- 
sures and higher on one). I Unlike the other skill areas, reading 

;V.domprehensi'on- was considerjed cumulative.. "Taught" here, was defined as 
"pass" beyond." If a curriculum unit containe'd passages of greater diffi- 

. culty, it was concluded that it "taught" passages of lesser difficulty. 
For .example, if a student had read a passage with a. vocabulary measure 
of 3.5, 'a sentence measufe of 10.3, and a passage length measure of 200, 
we assumed he would be able to read passages whose measures were lower 
than these, whether or not we could demonstrate that he had actually 
;read passages at that level. Thus, if we found that a reading curricu- 
lum of sixth grad^Uevel' was us'fed throughout the year, third grade items 
were considered covered as well. 

Four types of questions were associated with the test passages 
on the MAT Reading subtest. These were "main idea," "literal," "infer- 
•ential," and "word-in-context" questions . Again applying the concept- of 
:. .near trajnsf er, we dec?ided the curriculum fragment must include both a 
reading passage at thev.appropriate level and one or more of the four 
types , of que3tions .v-. . For example, if a curriculum passage was of an ap- 
propriate level for the Guy Fawkes reading item we displayed earlier as 
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Table- 6-1 



RANGES ON THREE INDICES FOR MATCHING REA'jING PASSAGES 
'IN PIP CURRICULUM WITH READING PASSAGES IN THE MAT 





Test 


Level 


Passage Feature 


Elementary 


Advanced 


Vocabulary range* 


-2.5 


-7.6 




■ 2.6-3.2 


7.7-8.1 




3.3-3.9 


8.2-8.8 




4.0-4.2 


8.9-9^1 




4.3-4.6 


9.2-10.7 




4.7-5.6 : 


10.8-11.5 




5.7-5.8 . 


11.5+ ■ 




5.9-6.4 






6.5+ • 




Mean sentence 


8.7-9.2 


12.2-12.6^ 


length (number of 


9.8-10.6 


12.7-14.4 


words in passage 


10.7-10.8 


14.5-14.8 " 


divided by number 


10.9-11.0 


14.9-.15.0 


of sentences in 


11.1-11.2 


15.1-16.3 


passage) 


11.3-13.1 


16.4-17.3 




13.2^-13.6 


17.4.18.5 . 




13.,7-14i8 


18.6-18.8 




14/9+ 


18.9+ 


Passage length. 


55-65 


105-149 


(in number of. 


66-77'. ' 


150-159 


Vords) 


78-79 


lj60-202 




80-82 


203-261 




83-89 


262-303 






304-313 






314+ 



Elementary level, Spach^/Dale-Chall; advanced 
level/ Dale-Chall. 



Exhibit G-l/and if that curriculum passage had only , inferential ques- 
tions, associated with it, we counted it a match only for Item 20 in 
Exhibit G-1. If a main 'idea' question was also associated with the 
passage, we, also counted a, match for Item 17. 

The other requirement for matching the reading comprehension 
"itemT"wTth"T"caFfrcuIum-m 
quired a student to read the curriculum /passages and the questions to 
himself. We felt that if a student use^ a tape with a book or confer- 
enced with the, teacher,' he would not have been giVen the skills needed 
to pass the test. / _ . 



Matheip.atics Computation Subtest 

■ 'Like the subtest -for wor/1 . knowledge, the Mathematics Comr ucc.- 
tioh subtest is quite straightforward. We reasoned that the math compu- 
tation items wobld probably appear vertically in almost the same manner m. 
the curriculum- the only variation would be' in the actual, value of the 
numbers. Our rules which were /cated as questions that had to be answered 
for both the curriculum item' arid the test item, specified the kinds of skil 
required for each item. ■ Among/ the ten math computation questions were the 
following: (1) What operatio^ is being performed? (2) What types of num- 
bers are being used? (3) Do^ the operation involve carrying? (4) Is the 
problem. written in the formybf an equation? - If the answers to the, ten 
questions were the same forVboth the test item ar.d the curriculum fragment, 
we counted the item as covered. , 



■ Mathematics Concepts Subtest 

The Mathemati/cs Concepts subtest was more difficult to analyze 
because each item dealfe with a nutaber of concepts at a fairly refined 
level. By examining publishers' outlines, which pointed out the skills 
that they were tryin/ to- emphasize, we developed criteria for matching 
the subskills required for each item. For example, in the measurement ■ 
items we asked what/basic operations students must use and whether the / 
students ware requi/red to convert the measure into another unit of mea- ^ 
sureC In the geor^try problems, our rules specified what shapes stu- 
dents were required to recognize, what geometric terms were used in the 
problem,' and whe4er plane or solid geometry was required.' With these 
levels of skills/ in mind, we searched the curriculum for exercises that 
would require only near transfer. 
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Mathematics Problem Solving Subtest 

For the Mathematics Problem Solving subtest/ we • accepted the 
.psychometric maxim that , the reading leval in the arithmetic story ppob- 
lems is deliberately set .low so that arithm<^tic rather than reading is 
being tested. However, each arithmetic story item was examined and, if 
any item contained words of low frequency for a grade level (Carroll 
Davies, and Richman, 1971), we eliminated it altogether froia our analy- 
sis. The rules for deciding that the remaining arithmetic items had 
bean taught^thus included the features pertinent to arithmetic opera- 
tions that we spoke of in the math computation rules, plus any special- 
ized arithmetic words used in the problem. 



Items That Could Not Be Analyzed 

0 

_ If we were unake to define clearly the skills needed for an- 

swering an item correctly, the item was dropped. . For example, we elimi- 
nated one type of arithmetic story problem, the one labeled by test pub- 
lishers as the multiple-step problem. The task requires several 
arithmetic operations , and sometimes a conversion of measurements: steps 
can b. performed, in different orders to obtain a solution, and the se- 
queace of skills is different for each pattern, of solution. Because-it 
vas too difficult to create a list of skill combinations that would per- 
mit literal matching with curriculum units, these a.-ithmetic problems 
were excluded from the evaluation. 

The only other items eliminated from the Ele-ientary and Ad- 
.vanced MAT battery were two . items in the Word Knowledge subtest. In 
.these Items the target word was actually a combination of three or four 
words. For example, Item'l9 on the Elementary battery was "A long 
wooden seat is a and Item 50 was "Snow piled by wind is a 

To get . each of these items . right, the student would have to encounter 
the same sequences . of words in his PIP materials. Because we could not 
expect to find a curriculum exercise that would provide/identical se- 
quences, we dropped these items. / 

■ ' / / . ^ 

Procedure s for Identifying Curriculum Materials 
th 3 t Match the Mt.T 

' . • 

A fui- evaluation of the PIP., required that their posttest effec- 
tiveaesE t.e judged only on those MAT test iterts that students should 
have passed, given the PIP-Specified curriculum materials that had been 
used. Our procedures for determining if the materials used in the 
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•fourth and eighth grades "taught^^ the skills required by the MAT items 
required a search . through the curriculum materials that were specified 
in the PIP and used in the projects x^7ith children in those grades. 

Another principle guiding our procedures was .that we did not want 
to claim that students should have correctly responded to a particular 
item unless we could show with reasonable certainty that, the curriculum 
materials contained lessons that would have ''taught^' the test item. Our 
rul-es for matching materials with test items were designed to be conser- 
vative^ allowing only for near. trans ferl*" We wanted to be certain that 
materials meeting our criteria.for corresponding with the MAT test items 
wot'ld permit the student to respond, correctly to those items. We were 
not particularly worried about ignoring test items thar had in fact 
been taught. But we were concerned about:- claiming that items had been 
taught i^7hen, in fact, they had not. Other . prof essionals might be less 
stringent and might assume, for instance, that lessons in advanced 
vocabularly would guarantee that less advanced vocabulary- was known. ^ 
We did not assume this* ' ; ^ " 



D ividing the Fourth and Ei-ghth Grad e: PIP Specified- 
and-Used Curriculum Materials Into Units . ■ 

■ ' To determine whether curjriculum materials contained lessons or 
ujiits that cc. formed, to the rules we had establ l^hed for each 'MAT .item^ 
'we'fevided each material ini:o "fragments.'* A fragment is a unit that 
deals^x>rith one skill. To identify a^ipropriate fragrrents^, we worked with 
both the SOIs and the materials: tor. each PlP-specified - and used mate- 
rial that was analyzable^ we looked at . each student assigmnent and then 
examined the material, to see how maqy distinct skills were covered in 
that assignment. If only one skill \<ras covered, we adopted the system 
the teacher had used. If more than one- skill was covered in the lesson^, 
we divided it into fragments so that only one skill per fragment was 
covered. , 

For the math miterials, \<!e found it relatively easy to, distin- 
guish the vario'j-j skills and separate the materials into fragments. The 
following examples are illustrative: . ; 
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Sullivan Mathematics Book-page Book 12, Page 3 

Singer Individualized Math Kit-block.-le^s-on---Kit: AA; Block 4, Lesson 12 
Systems. 80" , SeriSs-kit-lesson Learning Number Facts 

Kit Lesion 7 

For the word knowledge materials, we had to be more arbitrary. 
Because, we could not make each individual word a lesson, we depended 
more. on the organization set by the publisher, as illustrated below: 



Material 



SFA Reading Labs 



Systems 80 



Sullivan .Prograrnmed 
Reading 

McGraw-Hill Programmed 
Reading . . ' 

Random House Criterion. 
Referenced Reading 



Fragment 



Labs-colors- 
lessons 



Example 



Lab la^ blue^ Lesson 7 



Se.ries-kit-lesson Reading Wora^ in Context^ 

Kit Lesson 3 

Book-page Book 19, Page 90 



Book-page 

Level/skill- 
lesson 



Bodk 20, Page 45 
46-13 



Besides the series i. .terials, there were individual books, like Con- 
quests in Reading and Dr,_S£ello, that . included vocabulary words~They 
were generally broken down by page. . 



Like word knowledge materials, those for reading conprehefision 
were broken into fragment? . suggested by the structure the publishers had 
created, but the fragments were generally, larger because we made the as- 
sumption that reading is cumulative. 
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stories of the i aner City Stories 



Story 7' 



■B-ar-ne-1-1— Lof t-i^ ^eaifix:__ 
Skills So>ies 

McCall-Crabbs 



Random House Criterion 
Referencecl Reading 

Raiidora House Reading 
Series 

SRA Reading Lab 



- 

Skill-book-lesson • Main Idea^ Book 

Lesson 7 



Book-lesson 



McGrr..w-Hill Programmed Book 
IvPading - . 



Level/skill- 
lesson ^ 



Book Lesson 5 
Book 20 

48-5 



Level-difficulty Orange^ 5 
Lab-color ■ , Lab 'la,/-gold 



. Whenever SOIs failed to indicate how a material had been u^ed^ 
the site visitor asked the teacher. To -be sure that ''inateria' 7ere used 
as the publisher intended^ we asked if tapes had been used wit., the 
reading materials^ as expected in some kits^ and if teachers had pro- 
vided work sheets containing particular kinds of questions to test 
reading comprehension. . 
-V 

After determining -that enough students had used a. material^ 
that we could obtain a copy^ and that the material could. be broken down 
into lessons^ ~^we were in a position to analyze the fourth and eighth- 
grade curricu-la on a f ragment^by-f ragment basis. However^ we did not 
analyze alL material and ali fragments within a material because this 
was beyond, our res^ources. The next section describes the curriculum 
that we determined could be analyzed. ,^ 



Curriculum Materials That Could Be Analyze d 

Not all the materials specified by the PIP and used in the 
fourth and eighth grade classJrooms were analyzed. Table G-2 shows 
-which materials were' not selected for analysis and the reasons for their 
rejection. * . 

The least common of the five reasons for dropping a material 
•were^ first/ that the material was not used by. enough students to jus- . 
tify the time and- cost of analyzing it or, second^ that we were unable 
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Table G-2 

ACCEPTABILITY OF PIP SPKIFIEDtAND-USED MATERIALS FOR ANALYsfs 



a. Ca t c h - U p : Reading 




Scholastic Individualized Reading 
Language Master 

Criterion Reading, »^andcnj House 

Beginning to Read, Write, and Usten 

Correlated Language Arts, Open Court 

Reading Program, SRA " 

Reading Laboratory Kit— la, SRA 

Reading Laboracory Kits lb i Ic, SRA 

Systems .80 

Concept Development 
Learning Utter Sounds 
Reading Words in Co.ttext 

Barnell-Loft 

Getting the Facts 
Getting the Main Idej 
Drawing Conclusions 
Using thfe Confiext 
Working with S-^unds 
Detecting the Sequence 
Lo'cat ing th6 Answer 
Following Directions 



Yes 



Yes 



Yes 



4 
3 

1 
1 
1 



Yes 



Yes 



Yes 
Yes 
Yes 



4 
3 

1 
1 
1 



"Up: Math 



Material 



All Mathematics Subtests 



Analyzab le 



Hot Analyzable 



Sullivan Basal Mathematics 

Systems 80 

Learning^ Number Facts 
Developing Math Skills • 

Singer Individual ized .MdJChemat ics 

Tutor Computor 

Criterion Reference, Random House 



Yes 



Yes 
Y.>s 



Yes 



Yes 



1 



^ ^hl" T / Students; 2 - material unavail- 

able, indistinct lesson boundaries; 4 - inaporopriate 

format for the MtT; 5 - inappropriate skills for the MAT. 
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Table G-2 (Continued) 



c . Conquest : Ivead inr 





Word KnowlcdRe 


Keadinp Comprehens ion 


Materia 1 


Ana lyzab le 


riot 

Ana Ivzablc - 


Analyzable 


Uot 
AnalyzpGle 


Basic oipht Voc.dhulary Cards, Dolch 




3 




3 


Conquests in Reading 


Yes 






5 ■ 


Coronet Cassettes & Workbooks 




" I 




1 


Dr. Spello 


Yes 








Controlled Reader 




1 




3 


Merrill PhcVnics Skilltexts 








5 


Nicky 


Yes 




Yes 




McCal l-Crabbs Standard Lessons in Read in^ 




5 


Yes 




Primary Keadinf? Series, Bowmar 




I 




1" 


Propranmed Readiu)?, McGraw-HiU 


Yes 

a 




Yes 




llionovisual Wall Chart 




< 




5 


Kiiid , itudy , Think 




1 




1 


Kcjdinp Sk il r Builders , Reader's DiEest 




2 






Systems dO 

Heading Words in Context 
'•earning Letter Sounds 


Yes 


■> 




5 


Rcadinj^ Laboratory Kit la, SKA 


Yes 




Yes 




Tach i s toscope 




3 




3 


Uncle Bunny 




I 




1 


X..*rox Microfilm Read(«T 




3 




3 


barncll-Loft 

(,ctLii5£r the Kacts 

fsini' the CtinCrxt 

Lac all tip the Answer .; 

Workinp with Sounds 

Kol lowi ne- Di rrc t ions ■ 




5 
s 

1 


Yes 


3 
5 

5. 
5 



Kpv: 1 = material uaoil by ton few sriidrnLS ; 2 =^:i.itorial unavailable; 3— indisri-nct - 1 esaon 
boimdnritrft ; *♦ " inapprofir iate format ^for the MAT; 5 » inappropriate skills' for thti MAT. 



TatleG-2 (Concluded) 





d. HIT; 


Read ing 








Word Knowledge 


Reading ComDrehenaion 


Analyzable 


Ana lyzable 


Analyzable 


iVpt 


Conquests in Reading 


Yes 






Analyzable 


Remedial. .Reading- iTil Is 




5 




5 


Stories of fhe Inner City' 


Yes 




Ves 




Sulli\^an Reading' Prpgram . 


Yes 










• e. HIT:' 


Math 







MaterxRl 



Ideal Flashcards for Addition 
Self-Teacfiing Flashcards, Kenworthy 
Sullivan Basal Mathematics'' 



AH Mathematics- Subtests 
Analyzable I Not Analyzable 



Yfls 



f. IRIT; Reading. 
I Word Knowledge 



Ma t e r i a 1 



Mrs. Moon Series 

Programmed Reading, McGraw-Hill 

Heading Laboratory Kit l/i, SRA 

Heading Prapram, Hajidom Hopic 

tiarnel l-Lo(t 

Gotting the Fav.'ts 
Ciett infi^ the Main idt'a 
Locating the Answer 
Using the Context 



Analyzable 



Y.'s 
Yes 



Hot 
Ana ly'zgble 



Reading Comprehension 



Not 



Analyzable Analyzable 



Yea 
Yi»s 
Yt's 



Yes 

Y.}8 



Key. 1 - material used by too few students; 2 - material u:,ava i 1 ab 1 e ; 3 - in- 

H.^tinct le.,on boundaries; A - inappropriate format for the MAT; 5 - map- 
proprian» skills for the MAT. ^ 
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to get a copy of the. material to analyze. Eight of the materials were 
used by five students or fewer in all sites; for example^ the Coronet 
Cassette and Workbook was used by only three children in. one teacher's 
class. We were unable to obtain copies of two materials because the ma- 
terial was out-of-production (e.g.^ Tutor Computor)^ or the publisher 
would not lend a particular material that was too expensive to buy rela- ' 
tive to the data it would provide. 

More commonly^ a material was not selected for analysis" be- 
cause it did not have distinct lesson boundaries. Materials- such as 
f lashcairds^. gairies^ and teaching machines may cover (or not cover) a num- 
ber of skills, i-lepending on how they are used. For example^ a student's 
schedule might show that he was assigned a lesson on the Language Master, 
but~if^~the~ teacher" did not note which ski lls~ she was"w6f king "^^^ 
lesson, we could' not know which Language Master materials to include i'n . 
our analysis. In addition, because teachers might elect to make their 
ov/n cards to go with the Language Master machine, all we could know is 
'that the child had some sort of ' audiovisual lesson. 

■ ■ ^ . ' ■ . 

..The final two reasons for excluding a material from the analy- 
sis were that it. had an inappropriate format for the MAT or that it had 
inappropriate skills. -'Inappropriate format" meant that the student re- 
ceived help in reading the lesson; the MAT requires the student to read 
the items to himself. For example, the Mrs. Moon series has a tape for 
each book so the student can listen to the tape without reading the text. 
'^Inappropriate skills" were apparent in various degrees. The match 
could have been as far off as, "decoding skills" in the material and 
"word knowledge" skills on the MAT, or as close as reading comprehension 
vparagraph^ with the wrong kinds of questions. For example, the Specific- 
Skills series by Barnell-Loft has reading comprehension passages, but 
some of its programs lack iiteral or . inferential questions^ having ques- 
tions instead on sequence of events or on following directions. An exam-, 
pie of a complete mismatch with the MAT is. Rcmr^dial Heading Drills, which 
is used to emphasi/ic phonics; .the MAT Advanced has no items on phonics. 

Our exclusion of certain materials from the analysis does not 
imply that those curriculum materials are bad or that the MAT tests, the 
wrong skills. We were simply looking at what was PIP-specif ied and used 
in the classroom, and what parts of the MAT could be used to evaluate 
those {^articular programs. ^ 
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Sampling the -Reading Curriculum Materials • 

Except for reading comprehension materials, all PIP curricula 
that was specified, used, and analyzable (see Table 5-12) was examined 
for lessons that fit the rules we had written lor the MAT items. In 
other words, the universe of the PIP curricula/ was searched for material 
that matched, the. items, in the El^entary and Advanced Worcf Knowledge -sub- ■ 
test and the three math subtests. / 

■ Because of .fhe excessive number- of /reading comprehension mate- 
rials and the cost ^of analyzing each, some sampling procedures were devised 
The materials covered in projects where stud'ents took the reading sub- . 
„t?sts on the Elementary, and Advanced- Jevels .'of- the- MAT- were-selected-for^ 
sampling. The sampling procedure for the most frequently used materials 
at the fourth grade is described below: 

• SRA Reading Kit lA— The kit is divided into finely 
graded sections, each in a different color and each 
containing 12 cards. The last card' in each color, 
representing the most difficult, in each section, was 
used as the sample. ' 

• Random House Reading Series—Each of the two levels 

of books has 25 to 40 -books, 10 to 50 pages in length. 
Within each level the books are graded ,and marked by 
difficulty (ten gradations). The sample for each 
. book was 300 words drawn from the beginning, ^middle, 
and end of the book. 

• McCall-Crabbs, Book A— This book has about 30 short V ' 
readin^g passages, each aimed at slightly varying 

' reading l-v-^ils. -Therefore, any passage, indicated in 
. the SOIs as read, was analysed. 

Random House Criterion Referenced Reading— This se- 
ries is divided into five levels. Only the fourth 
level was analyzed; no pupil , used the 'fifth level, 
and the 'passages in the third level were very short 
(35 words) compared with the shortest MAT passage (57 
words on the Elementary). The fourth level contained 
about ten passages in each of two sections; five pas- 
sages frora each section were chosen randomly for 
analysis. - ; 

• Barnell-Loft— This series contains seven booklets for 
each skill: getting the facts, finding the main idea, 
and drawing conclusions. Within each skill, PIP 
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students read only the first five or sik bopklets, 

each of which contains 25 short pas,sages. Where 

length \<ias sufficient^ four passages were chosen from 

each booklet — one from the beginning, two from the 

middle^ and one from the end. 
■ ' "* ' ■ ■ ■ • ' / 

• McGraw-Hill Progranmied Reading — This;/ series has 21 

booklets^ each of 124 pages. Only Books 8-21 were 

analyzed; below Book 8 the passages/ consist of two 'pr 

three short sentences. In each booklet three or four 

tests are given to check mastery of preceding content. 

'I he longest story preceding each test was used as the 

i sample. When the story wa s too Ibnfi (10 pages), 200 

words from the m^iddle; of the scory were sampled. 

This material^ then^ was sampled most heavily because 

it is by far> the longest series ^and because it was 

frequently used iri several PIPs/. 

° The only ana.lyzable reading material used in the eighth grade 
was a 200-page book, divided, into approximately 20 stories^ each of 
xvhich was ^9,00 to'^SOOO words long. A sample of 225 words (nearest com-, 
plete sentence).^was drawn from the middle of each story. 

Results of the Curriculum S^fearch for Matches 
with MAT* Elementary and Advanced / 

Tables G-3 and G-4 display the results cfi)f match ing^ the descrip- 
tions of the MAT items with the PIP .materials • Two general comments can 
be made about '.the results of this analysis. First^ the skills needed 
for both the Reading and the Word Knowledge subtests were generally cov- 
ered somex^/here in the curriculum^ although we found that all PIP curric- 
ula analyzed- placed heavy emphasis on phonics, decoding, and word attack 
skills. These skills' are not. included in the MAT tests above Primer.. 
Second^ the' PIP curricula concentrated on ^asic operations presented in 
tl.e '!oI4 math*^ style; a fev; concepts were ''d iscussecl if they related to 
basic operations. The MAT divides its items into '*old" and "new" math 
and into basicand more advanced math. The MAT also places heavy empha- 
sis on understanding mathematical concepts.. 

• Elementary Word Knowledfte -rTablo . G~ 3a s;how.s the Word.s. 

used in the first three items of the Elementary Word Knowl- 
edge subtest. To ensure test security^ the remaining 4, 
items shown in the table include only the words "target^" 
"context,^' and "answer." 



G-20 



, ■ Table C-3 

CORRESPONDENCE BtrWEEN ELEMENTARY >!AT. ITEMS 
AND SPECIFIED-ANU-USED MATERIALS FOR ALL PIPS 



MAI ■ 
Icem Nunibcr 



Night 

IS ihe oppo - 
sice »t day 



Meac is a 
. . lood . 



Happy 
el ad 



TaraeC 
Answer 



Conquests 
in Reading 
(number of 

entries ) 



a. Word Knowledge Subtent 



Dr. Spello j 

(number of' * McCraw-Hi 1 1* 



entries ) 



(book» number 



15-21 

7-20- • 



Tarjjot * 
context 
Answ.'r 



1 jrw«'t 
AnnwtT 



■ T.ir.-.'t 



Tjr.'.ft 
l! o n t X t 
Ao'iw.'r 



T.irf?et 
Ans-vr 



l.ir,.-,.l 

Tjri;ot 
Anjiw** r 



15-17,20,21 
]fe-21 



Nicky 
'number of 
ent r ies ) 



Rar.djin House 

(nuaiber cf 
entries ) 



9-17 , 

v>-6''yMl,-15,lD,18 



16.17,:(> 

13, U,17-N.21 

11.13-^1 



i:. l *-21 
1?-U. Ih. I 7, IS 



s-io,i:.u.i7 



■i , h , •) . 1 1 J 2 



15-21 
15,10 

i3-i7,i<>,:i 



1 3-m 



-B, 1^-18 



I Ih.lQ 



./ , — 



i , 



An swiT 



An HWi' r 



Anaw»«r 



1 art'.jt 

Answi'i- 



.13-19 

10, i:-21 



17,10-^1 

Q>21 

12-1.., l'^ 



10. i: 



15-21 



Lnnr t'K t 
Ariswf r 



8-4:1 

!0. 1 i':o 



■ 10-1 19-21 

15-21 

tlj.l9 



nu,b..rs U.crd in.J,r«t. th- br.Us ,n wh.c. th. w..rd m cuver.-.! i,K t or ::«ore . 

;<•'. -i.^ dr'v.-lnp,'(} tor ilPfn. 



SRA 
\number of 
entries) 



5 



Systems 80 
(number of 
ent ries ) 



1 



Full 
Itti 
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EKLC 



/■ 

/ Tablt; C-3 (ContlnueJ) 



Word Knowledge Subtest (Continued) 



K/\ r 
71* Nunber 



:3. lar^L't 

Answer 



larcit 
Answ.fr 



Conquest s 
vn Ki'adin^: 
( nunbfr oi 

entr u-s ) 



Dr. Spello 
(number of 
en tries) 



:7. l.ir>j«?t 

Answer 



26. Target 

Context 



f ■ 



ERIC 



)0. IJ -L't 

AnKWer' 



31. Tarat-t 

An.su.* r 



McCrAw-Hill- 
(book number) 



Nicky • 
(number of 
••ntr ie« ) 



17-19. :i 
21 



•9. 12. 20. 21 



• 32. T.irvft 

An»-wi-r 



t .1 r 2 L" I 
Answer 



3^. Tar net 

Lonti.'Xt 
Answi-r 



Ltintt'xt 
Answi-r 



JO, 21 

7-lt3 

4 \ ,-2{\ 



9-U, lb, 17, 1*^.20 



Iir.-.t 
Answr r 



3^. i.iryt't 



IS, 1^- 



1*1-21 



Rondom House 
(number of 
entries ) 



SRA 
(number of 
entr ie» ) 



1 



20 

I J-l 'J 



20.21 

lS-21 

i:, li., i^. 



! — 



1 r If I ' r 
A nsui r 



■.I'-. T'.jriot 

A'nswi-f 



Ansu» r 



r ,4, 1 1 , 13. i^. 'I 
^,1V4^ 



Systems '80 
(i)umber of 
entr iei ) 



1 ' . 







i 




i 


Answer 


I 






, !. . 1 .1 r r 
















An su t» 1 












*' . 










■V'lsw,' r 










■ 






-iw .• r . 








TJT.-.'t 








^(■.<i t t xt 








.'.iibu. r 










1 ir I n .»{ i- t 








l.'vr I .M>»-'' 1 cr 


?• V< It 



J 2- 1. 21 



;i ^ 



t 1^ 
i i"i-:i 
: : f> , I h : 1 



1 • • : i 



Kull 
Item 
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TabU C.3 (Continued) 



/ 



MAT 
It^m Number 



^S. Target 

Answer 



Target 
Answer 



MAT 
Item 
Nuraber 



/ 



Of 

' 02 
0.1 
04 
Ob 
(Hi 
07 
08 
09 
lU 

1 : 

12 

n 

ii 
If' 
17 



SL 



i 3 



; .3 



13.6 



J..' I \ 
U 

34 ! j. 

.•^ 1 J 

i ^ 

J8 ! 3 

39 



40 
41 



43 
4^> 



7^> 



1 M 



I 14.8 



13. I 



Conquests 
in Keadini^ ' 
(number of 

entries ) 



a. Word Knowledge Subtest (Concluded') 



Dr. Spello 
'(number of 
entries ) 



McGraw-Hill 
(book number ) 



21 

20,21 



Nicky 
(number of 
entries ) 



'l^andom House 
(nuraber of 
entr ies ) 



SKA 
(nuober of 
tfnt r ies ) 



Systems SO 
(number of 
ient r ies ) 



Full 
Item 



■Spache 



3-9 



3.2 



Spache/ 
DC 



3.9 



■3.2 



S.3 



Random House 
ftead inR 



Blue Orange 
Level Level 



SRA 
Lab 

la. 



10 
10 
10 
10 
It) 

5,10 
5. 10 

5, IC 
5, 10 
5.10 



b- Reading Subtest 



McCal 1-Crabb« 
Book A 
( 1 L'ssnn) 



hlue 

» 1 ue 
Blu»? 
H I vie 



3,5,7,14 
5. ? 
5, ? 

3,5.7,1.'* 

5, 7 . 

3,5-7,9, 10,14 
■ 5-rJ.9,,lO 

5-7,9. 10 
5-7,9,10 
5-7,9,10 



Random House 
Cr iter ion 
Rdferenci> 
Level U 
Skill 8 - 
(lesson) 



3 

1.4 
1.4 

.4. 

l.-i' 4 

1,.',4 
1.2,4 
1,2,4 



QT • question typ*". 1 • m.in i,I,.a. 2 • word in 
AveijRe Spache and Oale-Chall. - 



Barnell-U)f t 
Drawing 
Conclusions 
(book) 



Barne 1 1 -Lett 
Cettinf. the 
Facts 
(book) 



E ' 



iiatnell-Lof t 
•Cetting the 
Main Idea 
(book) 



0,t 
A-K 



McCraw-Hill 
l*ro^.r.imm»»d 
Read infi 
(booki 



20. ?1. 

20,21 
20,21 

14-21 



1.1-21 
1(»-2I 



1 icera! , 4 ■ 



inft-rentisl ; PL - f.a^s^se l^nnb. SL • sentence length; 



Spache / DC 



„„,j,..r, u,t,d i„du.„. ,h. book, ,„ wh.ch th, w,.., i. „^ 
No rulej could be drveloped for this item. . " 
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Table G-3 (Concluded) 



c. M«the^'TlaC ica SubteaCs 





■ Mith 


ComputaC ion 




^ -n 


Math Coticepc* ; 






HaEh Problem Solvin 




j 


Systems 80 










Random 














Sull ivAn 


Learning 






Sullivan 


Sysletns BO 




Houne 






Sul 1 ivan 






MAT 




Numbur 


Singer 


Singer 


Bdsal 


Deye lopins 


Systems 60 


Criterion 


S inper 


Singer 


Basal 


Singer . 


Singer 




HaEhemaii.cs 


Facts . 


Kit AA' 


Kic QB 


Hathcmac ics 


MriCh Skill 


Preschool 


Reference 


Kit AA 


Kic BB 


MatheaaC ics 


Kit .AA 


Kic BB 


Number 


(bojk) 


(kit) 


(block) 


( b l,oc k ) 


(book ) 


(kit) 


(kit ) 


(-4cvcl) 


(block) 


(block) 


(book) 


(block) 


(block) 


01 


T ~ 


B , 


1. i 






: I «3 






> 


13 


4.8 




02 




A ' 


1.3 














6 


l\8 




03 






1.3 






j 








lb 


4,8 




04 


/16 




i.3 














14 


4,8 




05 . 


/ 


• 


6,8 




















06 


/ 16 
























07 


/ 1' 




■ •> 




















08 


/ IC 




5 










9 


6 








09 


/ ■ 1 >» 


D 


■ 2,3 




















10 




C 


2,3 










■ 










Jl 


/ 




6.8 




















1 2 


lb 




6,8 




■ * 


i 








* 






/ 


3b 
























14 / 


25 


E 




7 


















!V 


IB 




^8 




















lo/ 


3b 










i 




I- 








I'/ 


IH 




a .; 


















; U 
























r 
fir- 


» •'* • 
25 








3fj 


i * ■ 




9 












3b » 




V- 






AA 






9 








1 




24 




• *• 






















27 










j 


3,4 
































* 


* 


* 




?8 


















f ■ ' 






.. 26 


27 


G 








i 




9 








2 7 


36 










: I 














2fl 


18 . 




8 






i 1 • ■ ■ 














29 


\\ 










1 ■.. ■ 












■/ 30 


..•27 










; 1 














Jl 


24 • 










' ■ I 












* 


32 


25 


1 








1 1 




i 










33 


25 
























34 


3: 
























^b 


19 




8 






i ■ 1 




9, 


b 








36 


?9 
























,37 


11 








■ . ■ ■' .j 
















- 38 


























34 


33 
























AO 


22 




















o 





"^No rute.9 could developed for this item. 
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Table .G-4 

CORRESPONDLNCE BETWEEN ADVANCED NAT ITEWb 
AWD SPECIflEO-Ai-^D-USEU >IATERIALS FOR ALL PIPS 



Word Knowledge Suptest 



M/iT 
Item. Number 



Demolis hed 
means 
dest royed 



2. Used 

is Che oppo 
^^^^ ; new 

'To commenr 
• is to remark 

Target . 

Context 
Answer 

Target 
Context 
Answer 

. Target 

Context 
Answer 

Target 
Context' 
Answer 

Target 
Answer 

Target ' 
Answer 

Target 
Answer 

Target 
Answe r 

Target 
Answer 

Jarget 
Answer 




Stories of [ Conquests 

Inner City in Keading 

(number of (number of 

entries) | entries) 
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Table G-A (Continued) 



ci . ivord Knowlt?dro caibtest (Com innf.! ) 







bul livan j(:»kL) 


Stories o^t 
Inner City 


^ O n J i r L S 
in ivCl; ;i iv\^ 






item number 


book • 
huiDbe r 


i'Uinbe r i^ i 
Ln t r ies 


f iHiriber ot 
•nt r i^\s ) 


\ \ \\.\V.'.DV. T Oi. 

entrie.-: ) 


i'u 1 1 

i te:,i 


14. 


'larj^ec 
An swer 












. ■15. 


■rjr^:'et 




J - 










.Answer 












lb. 


Inreot 
Context 
Answei* 


6 


1 






1 


17 . 


Tar^'ot 
OOP text 
Ans we r 


^ 

J, 1 j 


1 

■ 2 








lb. 


'i r g e t 
Answer 


lu 


1 ■• 










i a r c e t 
■Con text 
Any we r 


6 


1 


1 


• 






"iaryut 
Context 
Ans we r 


• 




t 
1 

1 
\ i 






, 

22 . 


larijtet 
Answer 

I ar\?et 
Answer 






1 

■ 

/ 

/ 






zJ. 

j U . 


iarpet . 
Answer' 

1 ci r s» e t 

Answer , 








1 




23. 


iar^et 
Answer 












20. 


i.irpet 
Context 
Answer • 


b 
11 


1 
1 








2 i . 


.Ansvv' ■■; 






1 

! 


1 
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Tabid G-4 (Continued) 



a, word Knowledge Subtest (Cone inued; 



MAT 
Item Number 



26, 



29, 



Tarj>et 
Context 
Answer 

Target . 
Answer 



30. 



Jl . 



33. 



34. 



33. 



36. 



37. 



Jo: 



40. 



Target 
Context 
Answer 

Target 
Context 
Answer 

T<arget 
Context 
Answer 

Target 
Answer 

Target 
Answer 

Target 
Answer 

Tareet 
Context . 
Answer 

Ta.rf^et 
Context 
Answer 

Target 
Answer 

Target. 
Context 
An.s?wor 

Target 
Answor 

Target 
Answer 



/ 

Sullivan (bKL). 



book 
I'Jumber 



8 
25 



12 



25 



Number of 
Lnt r ies 



Stor i.'s oi 
i nnor L> i ty 
(riut:iber jl 
entri :o j 



Contjue.'it s 
in AiMdinr 
( i:uiiiL)i? r 6 r 

entries ) 



b ul i 
ltd;-. 



':;};■ 
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Tabid G-4 (Continued) 



Word Knowledge Subtest (Concluded) 



MAT 
Item Number 



42. 



Target 
•Context 
. An s w er 



43. 



Target 
Answer 



A4. Target 

Context 
Answer 

45. Target 

Context ^ 
Answer 

46. Target 

Context 
Answer 

47. Tars^et 

* Context 
Answer 

48. Tar^et 

Answer 

49 . Tareet 

• Answer-' 

50. Tarr.cit 

Context 
Answer 



Sullivan (BRL) 



Book 
Number 



17 



Number ' of 
Entries 



.Stories of 
Inner City 
(number of 
entries ) 



Conquests 
in Reading 
(number of 
entries ) 



Full 
Ltem 
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Table G-^ (Continued) 



MAI 
Item 
Number 



01 
02 
03 
OA 
05 
06 



— err 

08 
09 
To- 
ll 
12 
1-3 

15 
16 
17 
18 

.19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

2y 

30 

31 

32 

33 

3A 

35 

36 

37 ■ 

3d. 
39 
AO 
Al 
A2 
43 
AA 
A5 



b> Reading Subtest 



Question 



A 
2 
3 
1 
A 
1 



— 2" 
A 
A 
A 
A 

a' 

' A 

2 

4 

4 

3 

3 
•-4 

2 

- 2 
I 

A 
A 
A 
3 
3 
2 
2 
2 
A 
1 
3 
A 
2 
2 
3 
4 

4 
4 
4 
4 
2 
4 



Passage 
Length 



104 



148 



205 



Sentence 
Length 



17.3 . 



16.4 



15.6 



J59 



260 



302 



i5.9 



16.5 



16.8 



316 



15.0 10.5 



Dale- 
Chall 



Stories of 
Inner City 
(pages) 



8.8 



7.5 214-218 



8.1 



214-218 
214-2r8 



lb. 



11.5 



9.1 



1 = main idea. 2 = word in context. 3 = literal 
4 = inferential . * 
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Tflble G-4 (Conclure'd) 



Mathematics Subtests 



Math Computation 



MAT 
Item 
Uumber 



01 
02 
03 

03"" 

06 

07 

08 

09 

10 

11 

12 
13 
lA 
15 

17 . 

18 

19 

20 

2d 

22 

23 . 

24 
••2b 

26 

27 

26 

29 

3U 
31 
32 
33 
34 
33 
3b 
37 
38 
39 
40 



Sul livan 
basal 
Mathemat ics 
(book) 



24 
11 
27 
21 
31 

16 
20 

IS 

36 

29 

31 

23 

27 • 

36 

31 

36 
32 
30 

32 
23 

32 



32 
32 ■ 
3o 
32 
3o 

36 



33 



Ma t h Concepts 



MAI 
Item 
•^uiTibor 



•01 
02 
03 
04 
03 
06 
07 
08 
09 . 

id ■ 
11 

12 
13 
■14 
15 

,16. 
'l7 

18 

19 " 
.20 

21 

22 

23 

24 

25 

26 

2 7 

26 

29 . 

30 

31 
32 
33 
34 
35 
36 
37 
38 
39 

'-40 



Sullivan 
Basal 
Mathematics 
(book) 



36 



31 



36 



Math Problem Solving 



MAT . 
Item 
Number 



01 
02 
03 
04 
05 
06 
07 
08 
09 

10' : 

11 
12 

13 ■. 
■ 1,4- 
15"' 
16 
17- 
18 
,19 
.20 
21 
22 
23 
24 
25 
26 

27 
28'- 
29 
30 
31 
' 32 
33 

35' 



Sullivan 
Basal 
Mathematics 
book) 



24. 
28 



36 



32 



36 



'no rules could bc! developed for this itcni. 
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Some of the items do not list the context word, which 
means that the context word.was "is" or "is to.^'^' ■ y/i re- 
. viewed the elementary literature and found that "is" was 
so commonly used that it was not necessary to check it for 
each individual item. The numbers under ec2ch material 
show the number- of times that a particular word was found 
in that material. Book numbers are'shOwn for the McGraw- 
Hill series, however, so that the reader may see how the 
words are distributed across, the 21 books.* 

While many of the MAT words were found in the materials, 
the data base. for our ^finest level of analysis was re- ' 
stricted to MAT items for which each- of the three wbi^s- 
were found. These items are labeled "Full Item" in the 
right-hand column. - Relative to Item. 8, for example, the 
word "tent" was found, but the other two (teepee, kind) 
were not. Although' about 877» of the MAT words were found 
in the materials,, only 56% of the found words were part of 
full items. \ ■ 

Elementary Reading-Table G-3b shows the correspondence 
between the Elementary Reading subtest and the materials. 
The question type (QT) and passage features for each story- 
are listed for each item. 

The beginning levels of the materials [SRA, Random House - 
Criterion Reading, McGraw-Hill, and Barnell-Loft (Getting 
the Main Idea and Drawing Conclusions)] did not contain 
passages of sufficient length to meet the requirements of 
the test's readability analysis (vocabulary, mean sentence 
length, and.- passage length). ' Few materials lacked the 
requisite types of questions, with the notable exception* 
of word-in-context. The PIP fourth and eighth grade mate- 
rials emphasized literal and inferential questions. 

Since the Elementary Reading subtest was given at pre- and 
post-test to fourth graders, the passage features of the ' 
curriculum materials cluster about the items at or below 
the fourth grade level. Thus, these materials do not 
match about 60% of "the test items. 



The book numbers listed on Table G-3a indicate the books in which a 
particul,ar._word is covered six times or more. 
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E lementary MarhPrnafics Coinputation - -T&ble G-3c shows^the . 
correspondence between the materials and all three of the 
Elementary math subtests. An asterisk indicates the items 

• for which we could not develop rules. All items in the 
Elementary Mathematics Computation subtest were founej in 
the curriculum materials. Basic operations are emphasized 
in the test and in Catch-Up. 

Elementary Mathematics Concep-.s -While the materials cov- 
ered approximately 18% of the items on this subtest, con- 
gruence ,is spotty and limited to part of a Sullivan book- 
let, one Systems 80 card, and a few pages in a reading 
series. , v ' ■■ 

. v^lPrrPnfAr y Mathematics Problem Solving -Few of the PIP laa- 
■ terials in Cntch-Up contain story problems like those in 
this subtest. The S'ullivan series contains none at all. - 
We considered. only the most simple problems (basic opera- 
tions) to be candidates for correct responses. 
.. AHv.nrP.l Word Knowledge -rTable G-4a lists the word knowl- 
edge items' for the Advanced MAT and the , corresponding les- 
sons in the PIP materials. The. number of MAT items found 
in the materials were far fewer than -for the El T.entary 
test While this difference can be partly attributed to 
■ the smaller number of materials in KIT,. the xiif f erence is 
"probably also due to difficulties in sampling the larger 

• vocabulary of eighth graders. Even for the two iteihs cov- 
ered in toto, the: target word for Item 1 was found on only 
one page and the correct response on only one page. Ap- 

.,. proximately . 50% of the words-either target, context, or 

answer— were not found in the materials. 
• ^ Advanced Reading- Table. G-^b shows that only one material, 

Stories of the Inner City, contained questions and pas- 
■■. sages similar to those on the test.; Whil^ many stories 
" 'had the appropriate questions, the. passage feature indices 

were considerably lower than those of 90% of the test 

items. ■ s • 

Advan ced Math »n.;,fi r.s Comoutation-The only mace rial ana- 
lyzed for math subtests was the Sullivan Basal Mathematics 
• seri-^s (see Table G-4c) . As in the fourth gr^de, basic 
operations were emphasized in the eighth grade, but the 
Advanced MAT included more highter-level math problems. 
This series matched about 70% of the test items. 
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' Advanced Mathematics Concepi-c;„ThP Sullivan series con- 
X tains few concepts other than those pertinent to basic op- 
erations on whole numbers, fractions, :.nd decimals. The 
Mathematics Concepts subtest consists .of many items (about 
957o) that cover skills/ now labele-^ new math, that are not 
covered in the eighth grade curriculum. 

' .Advanced Mathematics Problem ,Sntv^r^--Tv.o Sullivan series 
does not conta^in any story problems similar to those in 
this subtest. Consequently^ only the most simple problems 
(basic opetkt ions) were considered to be candidates for 
learning due to TIP instruction. 

In..reviewing this comparison ot the MAT ' and tne PIP-specif ied and 
u^ed. curriculum, the reader is reminded that we'were able to look only 
at the materials kno^wn to be covered from January to posttesting. We 
know nothing about assignments from pretesting to January, nor do we 
have concrete .' nformation about lessons assigned in the 'regular curric- • 
ulum. Although we are confident that the SOIs reported most of the PIP 
treatment for students in the fourth and eighth grades from January to 
posttesting, we cannot guarantee the inclusion of all lessons assigned. 
Some of the MAT items we have. excluded from the analysis might be rele- 
vant to materials we have, not examined. Possibly the MAT 'is more rele- 
^vant to the fourth md eighth grade PIP curricula than.it appears from 
our analysis, but :.j,e are limited to conclusions that can be drawn from 
the assigjjiments reported on the SOIs. ■• 
., ' ' ' ' " ' ' 

Table G-5 displays those MAT item numbers for each subtest in the 
Elementary and Advanced battery that were covered by at least one frag-' 
™«"t jised in the fourth or eighth grade. "N" represents the greatest 
. number of it^m. in our analysis any student could have answered correctly 
If he covered, all of the PIP-specif ied materials used in the projects. ' 
Only la Math Computation does the MAT appear relevant to the fourth and 
eighth grade curriculum. That the projects with a mathematics component 
do well on this MAT. subtest is probably because of the greater relevance 
of this part of the MAT to the projects ' 'objectives. 

Attempts to Match Eac h Student with the Items He Had Covere.d " 

^ As nor d earlier, we planned to analyze the relevance-rof the MAT to 
the curriculum at two levels.. We have described the intermediate level 
in the previous sections. In this section we describe the more detailed 
level, where we, matched each student with the^ items he covered. 




Tabid G-5 

THE MAT ITEMS KNOWN 'TO BE COVERED BY PIP-SPECIFIED MATERIALS 



•a. Elementary Battery 



Word 




Mathemat ics 


Knowledge 


Reading; ; 


Computation 


Concepts 


Prob lem Solving 


i 


i 


1 


1 


1 




0 

Z 




o 

• • 8 


*> 
z 


J 


J. 


3 


zU 


J 




/, 

H 


■ 4 


zl 


/. ' 


c 
J 


C' 

-> 


5 


2 J 


10 


6 


0 


6 


2d 


0 tl 

Z3 




7 


7 


35 


. T = J5 ^ 


Q 


Q 
O 


'8 


T = 40 


M — iiL, 
N — 0, 


i J 


Q 


9 




N ■ ^ 1 / . i ^ Q J 1 


I 0 


1 n 

iU 


10 


N - i / . j/» or i 




i / 


1 1 ■ 
I i 


1 1 






1 Q 
i O 


1 0 
i. z 


12 • 






00 


■1 A 


13 






^-^ . 


i-> 


14 






O Q 


1 0 


15 






JU 


i 0 


lo 






T 1 


z u 


17 






Q/i 


. I — Hj 


lo 








' M — 17 

, IN — i / 


19 






SI 


, N = 37.7% of T 


20 






38 • 




21. 






39 




22 






41 




.2-3 






42 




24 




■ ■ 


45 




25 






46 


r 


26 . 






47 




' 27 ^ 






49 




2:8;V 






T = 50 




29 










30 






N = 56% of T 

i ■ 


i 


31. • 

33 
34 
35 
36 
37 
38 
39 
40 

. T = 40 


s 








N = 40 
N = 100%- of T 







Key:'- I =" total number of items in MAX subtest; N = number of MAT items known 
to; be covered by PIP-specif ied materials. > 
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Tabid G-5 (Concluded) 



Word 
Know I edge 



b. Advanced ' Battery 



1 

5 

T = 50 
N = 2 
N « L^t of T 



Reading 



W = 



5 
8 
9 

T = 45 
N - 3 
6.6% of T 



/ 



Computa t ion 



1 , 
2 
3 
4 
5 
6 
7 
8 
9 

10 ' 
11 
12 
14 
15- 
16 
18 
19 
20 
22 
23 
24 
27. 
. 28 
29 
30 
31 

" 33 
36 ■ 
T = 40 • 
. N = 28 
N - 70% of T 



Mat hematics 
' Concepts 



1 
4 
2D 
T = 40 
N = 3 
7,5% of T 



Problem Sol V in p 



T 
N 

N = 14 



• 1 ■ 

2 

6 
12 
24 

=~35 . 
= 5- 
.2% of T 



Jn\p°''^ ""f ofo''"""^'.'" subtest; N = number of MAT items known 

to 4>e covered by PIP-specified materials , 



4i i 



The methodology for the detailed analysis required returning to the 
SOIs and recording each studeht's PIP-specified 'assignments. We did this 
for students who took either the Elementary or the Advanced' MAT and whose 
teachers had either a high or low (5, 6, 8, or ,9) implementation rating. 
Exhibit G-2 displays the form used to inventory the PlP-specified. assign- 
ments for a student in Conquest. These ihventgry forms . included all • 
materials analyzed in Section 5.7. ^ ' 

The next three steps (already complfeted in the" intermediate- level, 
analysis) were: (1) analyzing the Elementary and the Advarced MAT items . 
for the skills needed for the appropriate answers ^described in Section 
5.7), (2) analyzing the curriculum to finrf the- lessons that taught tho^e 
skills, and (3) developing a list (Tables 0-3 and G-4) of the MAT items 
that had been covered by some part of the PIP" specif ied-and- used cur- 
riculum. ' . 

The final step required determining the' correspondence between each 
student's inventory of PIP assignments and the MAT litems covered by those 
assignments. This was done by overlaying each" student ' s. inventory of 
PIP assignments on the list of .MAT items covered by the curriculum materials. 
(Tables G-3 and G-4) . To determine whether'a student had passed the 
items he had covered (that he should haye answered ^ correc t ly) , we planned 
-to sort each student' s file as illustrated by Che following schematic r,^ 



MAT Items 





. » • 


1 


2 


3 


4 " 


•5 


6 


7 


8 


9 


10 


Student' 


s pretest 


0 


0 


0 


' .Q 


1 


1, 


1 


1 


1 


0 


Covered 


by PtP curriculum 


0 


1 


0 


1 


0 


. 1 . 


0 


1 


0 


.■ 1 


Student' 


s posttcst ■ 


■- .0 


1 


1. 


.0 


0 


0 ■■ 


1 


1 


1 ■■ 


1 



'A "O" or a *'l*' for the pre- or the post-test indicates a child's failure ^ 
or success-, . respectively, to correctly answer a particular MAT item; a . 
**0*' or a "l" for the "covered by PIP curriculum'* indicates the student's 
lack of exposure br his exposure to materials related to that .MAT item. 

From the outset we anticipated having t'oo few data points (students 
and materials/MAT correspondences) to warrant completing the final sort 
into correc t- incorrect responses or performing the analysis. After we 
had determined which, it^ms each student had dover^ed, it became apparent 
that there wera indeed; two few. data points. ■', 



Table G-6 displays t'he results of the last step we completed. The 
•number of students who had covered at least one MAT item in their curricula 



: Exhibit ' 

INVENTORY OF INDIVIDUAL STDDEW'S PIP-SPECIFIED ASSIGNHE^S 



Werfiflj -tilt Sejgentt 
Of. ifidir, f 
' 33 • JoV ' 



so 
51 

ss 



• J' 

'• 33 
•■ 1< 



lot 





Coft(?U!^l'-'Rea(li(W 



I 



G<i!i. 



B 
C 

unit 1 
. II 

uflrt a.. 
Bwk F ■ 



Hoeb 



✓ ! 

al0r 



HiffiS 



8iip.3-;i 



Table C-6 



/ Xn-IHKR OK ITEMS COVERED BY STUDENTS 

' . IX hiACH PIP': GRADE 4 or GRADE' 8 







Elementary MAT 


Advanced MAT 






Cn r- rh-llo 


Co noue s t 


IRIT 


■ HIT 


(A) 


^ / ■ 

/ 

Number of ^;t\idents in grades 4 or 8 'whose • ■ 
teachers were rated as goqd or bad 


■ 27 


30 


28 


63/40'' 


• Total 'Rending 










(B) 


Number of students in (A) who covered one or 
more , items on the MAT in their individual 
reading curricula ' ' . 


8 


25 


23 


I 


(C) 


Kuiober -af reading itens possible for . 

students in. (B) . 

/ ' ■ ■ . 


360 


1 125 


1035 


5 


(D) 

ft ■ 


Number of reading,' items covered by students 
in (B) ' . \ 


9. 


129 


227 




. / ''L^ Total Math 










j (eV 

t 

t ■ / 

i 


"^Number of* student's In (AVwho cohered one or 
more items on the MAT in their individual 
math curricula 


1 ' a 


NA 


1 NA 


, 34 ' 


(p) 


Number of math items possible for students 
in (E) ' . ■ . •■ -.: 


1 

. 1013 


' ' NA^ , 


: NA 

i 


1178- 


i /G) 

I ' 


Number 6f math^ items coveted by students, 
.in (E) ' ) ■ " 


18 9. 


^ NA . 


NA 


1 

162 ■ 


XA = 


■ . ■ ■ ■ ■ S' ; ' 

N'ot applicable. ' ' ■■■'*•..■ 











Th'er.e were 63 students in the HIT reading program and 40 students ^icv;>;.!cfe^Hr 

+ . ■ ■. ■ • . ' ' . , i . ■ .'; ■ '''. ■'':' "" ■' 0 

The- nuitiber of possible items for all students in Group (B) was r.^al-C.ul'ated by mult iplyintj 

Ar. ; thcv number of possible items (Table '5-15) on each subtest by' t He ■'■p'um o'er.- of students with 
vXt-lid scores on that subtest-. . . • • 'ii-. , •< . • . 
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was quite small-much- smaller 'than .the number being considered in our 
fourth and eighth grade sample. Moreover, the number of items actually 
covered, when compared with the nc^ber of possible items, was again 
quite small. This was especially o.vious in light of our plans, which 
called for analyzing data within project by comparing^ the scores of 
children who had well-implemented/responsive teach,ers with the scores 
of children who had poorly implemented/nonresponsiyeVteachers. With so 
few. items there did not appear to be enough data in each category to 
analyze effects. , . & j ^ 
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